Word Database Research Articles

A Bi-LSTM and Attention Based Approach for Developing an Assamese AI Chatbot

Objectives: The main objective of this study is to develop, evaluate, and analyses a Bi-LSTM and attention-based model and decide whether it is the best deep learning hybrid model for developing the Assamese AI Conversational Agent. Methods: A dataset for the Assamese language was created for this research. A bi-LSTM and attention-based model was developed and trained with the dataset. We have compared the accuracy and performance of the proposed model with those of other deep learning models. The bag of words technique was used for feature extraction of text. For natural language processing, a custom stemmer, stop word functions and root word and stop word database were developed. The performance of the model is measured using metrics such as precision, recall, and F1-score. Findings: From the research study, we have found that the bi-LSTM and attention-based model performed better in terms of all the metrics compared to other models. This model was able to attain an accuracy of 89.99%. Novelty: This research is novel as it is the first attempt to develop an AI chatbot in Assamese based on a deep learning model. Assamese is a language that has not received much attention in the field of AI chatbots. This research has great significance for the society where Assamese is spoken, as it can provide a platform for communication, information, and education in their native language. The Assamese language is in its beginning stages on the digital platform and has limited research done in the field of AI chatbots. This research will contribute a lot to this research area and open a new door for researchers who are interested in this topic. It will also allow them to further explore, improve, and develop new ideas. Keywords: Deep Learning, Natural language processing, Chatbot, Assamese

Taboo language across the globe: A multi-lab study

The use of taboo words represents one of the most common and arguably universal linguistic behaviors, fulfilling a wide range of psychological and social functions. However, in the scientific literature, taboo language is poorly characterized, and how it is realized in different languages and populations remains largely unexplored. Here we provide a database of taboo words, collected from different linguistic communities (Study 1, N = 1046), along with their speaker-centered semantic characterization (Study 2, N = 455 for each of six rating dimensions), covering 13 languages and 17 countries from all five permanently inhabited continents. Our results show that, in all languages, taboo words are mainly characterized by extremely low valence and high arousal, and very low written frequency. However, a significant amount of cross-country variability in words’ tabooness and offensiveness proves the importance of community-specific sociocultural knowledge in the study of taboo language.

The implementation of truth commission recommendations: Exploring the ‘beyond words’ database for Latin America

We know little about the processes surrounding the formulation and implementation of truth commission recommendations (TCRs). This article introduces the Beyond Words database, which systematically identifies and tracks the formulation and implementation processes behind 960 recommendations from 13 truth commissions established across Latin America between 1983 and 2012. The database offers information on factors shaping TCR formulation, identifies important characteristics of recommendations and assesses the level of implementation for each. We describe our methodology and the resultant database and examine findings linking TCR characteristics and contextual factors to levels of implementation. Overall, the implementation record is better than those previously considered.

A study of feature extraction for Arabic calligraphy characters recognition

Optical character recognition (OCR) is one of the widely used pattern recognition systems. However, the research on ancient Arabic writing recognition has suffered from a lack of interest for decades, despite the availability of thousands of historical documents. One of the reasons for this lack of interest is the absence of a standard dataset, which is fundamental for building and evaluating an OCR system. In 2022, we published a database of ancient Arabic words as the only public dataset of characters written in Al-Mojawhar Moroccan calligraphy. Therefore, such a database needs to be studied and evaluated. In this paper, we explored the proposed database and investigated the recognition of Al-Mojawhar Arabic characters. We studied feature extraction by using the most popular descriptors used in Arabic OCR. The studied descriptors were associated with different machine learning classifiers to build recognition models and verify their performance. In order to compare the learned and handcrafted features on the proposed dataset, we proposed a deep convolutional neural network for character recognition. Regarding the complexity of the character shapes, the results obtained were very promising, especially by using the convolutional neural network model, which gave the highest accuracy score.

Chinese Natural Language Processing: From Text Categorization to Machine Translation

Abstract The level and volume of automatic computerized processing of linguistic information has become one of the most important criteria for measuring whether a country has entered the information society. The study begins with statistical linguistics and aims to process complicated Chinese information. In this paper, after establishing the word database of the Chinese language, the language model is smoothed and compressed, the Chinese character information and Chinese language information are extracted, and the processing of Chinese grammar and Chinese semantic information is emphasized. Among them, Chinese grammar processing includes Chinese word analysis and basic phrase analysis based on the maximum entropy model, and Chinese semantic processing includes Bayesian-based word sense disambiguation, semantic role labeling based on the conditional random field model, and thesaurus-based semantic similarity calculation method. In addition, SECTILE-based Chinese text categorization and statistical linguistics-based machine translation methods are explored and tested for their effectiveness in Chinese natural language processing. The results show that the total average check accuracy and check the completeness of Chinese text are 78.65% and 72.24%, respectively, and the BLEU values of the translation methods are improved by [1.62,3.73] and [0.93,5.01] compared with the Baseline method, which is able to process the Chinese information accurately. The research plays an important role in the process of information processing based on Chinese language processing.

Emotional Prototypicality Ratings for 636 Chinese Words: A Database of Chinese Words with Affective Information

Exemplars of concepts vary in their degree of prototypicality. This is also true for emotion concepts. This study presents prototypicality ratings for a large set of Chinese words. The database contains 636 potential Chinese emotion words (i.e., words that directly express particular emotions, like “高兴 happy” and “哀愁 sad”), from different grammatical categories. Native Chinese speakers rated the words in terms of emotional prototypicality. The database also contains values for valence, arousal, and emotionality. The analyses of the ratings revealed that 502 out of 636 words had a high prototypicality value (value equal to or above three on a 1-to-5 scale), the most prototypical words being negative and high-arousal words. The analyses also indicated that the emotional prototypicality of a word was positively related to both arousal and emotionality, and negatively related to valence. Among these variables, arousal was the most important contributor. Similar results have been found in studies conducted in other languages. This will be a useful resource for researchers interested in studying emotion words in the Chinese language and for those interested in cross-linguistic comparisons.

A large-scale database of Chinese characters and words collected from elementary school textbooks.

Lexical databases are essential tools for studies on language processing and acquisition. Most previous Chinese lexical databases have focused on materials for adults, yet little is known about reading materials for children and how lexical properties from these materials affect children's reading comprehension. In the present study, we provided the first large database of 2999 Chinese characters and 2182 words collected from the official textbooks recently issued by the Ministry of Education (MOE) of the People's Republic of China for most elementary schools in Mainland China, as well as norms from both school-aged children and adults. The database incorporates key orthographic, phonological, and semantic factors from these lexical units. A word-naming task was used to investigate the effects of these factors in character and word processing in both adults and children. The results suggest that: (1) as the grade level increases, visual complexity of those characters and words increases whereas semantic richness and frequency decreases; (2) the effects of lexical predictors on processing both characters and words vary across children and adults; (3) the effect of age of acquisition shows different patterns on character and word-naming performance. The database is available on Open Science Framework (OSF) ( https://osf.io/ynk8c/?view_only=5186bd68549340bd923e9b6531d2c820 ) for future studies on Chinese language development.

In the melting pot of web‐crawled texts: The challenges of extracting English words from Croatian corpora

AbstractThe focus of this paper are English words and phrases used in Croatian which, unlike loanwords, have not undergone major adaptations at the orthographic, phonetic, or other levels apart from being influenced by the inflectional system of the recipient language. A list of English words in Croatian corpora was compiled using automatic algorithm extraction, corpus query language in Sketch Engine, and manual word list evaluation with the end goal of publishing the first comprehensive online database of English words in Croatian. The ENGRI corpus of Croatian was created by web crawling procedure and used together with the existing Croatian hrWaC 2.2 RFTagger corpus to produce a list of English words and phrases. In this paper, word list compilation issues are discussed in relation to both general issues encountered in the study of interlingual lexical types (such as false cognates, antonomasia, and polysemy) as well as Croatian‐specific language properties such as its inflectional system and diacritical marks. In conclusion, we propose that manual evaluation is an indispensable method and a necessary complement to computational linguistic tools in the creation of word lists and databases of foreign words in other languages.

The Pandemic in Words: Tracking Fast Semantic Changes via a Large-Scale Word Association Task.

Most words have a variety of senses that can be added, removed, or altered over time. Understanding how they change across different contexts and time periods is crucial for revealing the role of language in social and cultural evolution. In this study we aimed to explore the collective changes in the mental lexicon as a consequence of the COVID-19 pandemic. We performed a large-scale word association experiment in Rioplatense Spanish. The data were obtained in December 2020, and compared with responses previously obtained from the Small World of Words database (SWOW-RP, Cabana et al., 2023). Three different word-association measures detected changes in a word's mental representation from Precovid to Covid. First, significantly more new associations appeared for a set of pandemic-related words. These new associations can be interpreted as incorporating new senses. For example, the word 'isolated' incorporated direct associations with 'coronavirus' and 'quarantine'. Second, when analyzing the distribution of responses, we observed a greater Kullback-Leibler divergence (i.e., relative entropy) between the Precovid and Covid periods for pandemic words. Thus, some words (e.g., 'protocol', or 'virtual') changed their overall association patterns due to the COVID-19 pandemic. Finally, using semantic similarity analysis, we evaluated the changes between the Precovid and Covid periods for each cue word's nearest neighbors and the changes in their similarity to certain word senses. We found a larger diachronic difference for pandemic cues where polysemic words like 'immunity' or 'trial' increased their similarity to sanitary/health words during the Covid period. We propose that this novel methodology can be expanded to other scenarios of fast diachronic semantic changes.

TWITTER SENTIMENT ANALYSIS

Sentiment analysis is a popular research field that aims to extract opinions, attitudes, and emotions from social media platforms like Twitter. Traditionally, sentiment analysis has focused on analyzing textual data. Twitter, being a well-known microblogging platform where users post updates in the form of tweets, is often used as a data source for sentiment analysis studies. In this particular paper, the authors utilize a publicly available labeled dataset from Kaggle. They also propose a set of preprocessing steps to enhance the manageability of tweets for natural language processing techniques. Since the dataset consists of pairs of tweets and their corresponding sentiment labels, supervised machine learning is employed. The authors propose sentiment analysis models based on naive Bayes, logistic regression, and support vector machine algorithms with the objective of achieving more effective sentiment analysis. In Twitter sentiment analysis, tweets are typically categorized into positive or negative sentiment. Machine learning classifiers can be utilized for this purpose. These classifiers can be valuable for various entities such as businesses, political parties, and analysts, as they can evaluate the sentiments expressed towards them. By appropriately training the machine learning models with labeled data, tweets can be accurately classified without the need for a predefined word database. Consequently, machine learning techniques offer a superior and faster approach to performing sentiment analysis. Keywords: Twitter; sentiment; Web data; text mining; SVM; Bayesian algorithm; hybrid; ensembles

Text Vectorization Techniques Based on Wordnet

Abstract The utilization of text vectorization techniques has become essential for numerous classification tasks in present-day natural language processing. Word embedding methods commonly used today, such as Word2Vec, GloVe, etc., are based on the semantic similarity of words. WordNet, as a lexical database of words, provides a rich source of semantic information. In our article, we propose a text vectorization technique using extended text data with the data augmentation method, specifically by replacing words with their synonyms obtained from WordNet. The results obtained from text classification tasks using multiple classifiers demonstrate that expanding the corpus with this method leads to improved vector representations of words.

Differences in young children's emotional valence ratings of 180 stimuli

Words and pictures are often used as stimuli in studies of investigate relations between emotion and cognition. Selecting controlled stimuli for experimental design in this area represents a crucial issue. Only few standardized databases of pictures or words are available for children. None compile at the same time ratings for both lexical and visual stimuli. The present study provides a French child database for a corpus of 180 stimuli (90 words, 90 pictures, 90 combined words-pictures). These stimuli were rated on emotional valence (emotional: negative, positive, and non-emotional: neutral) by 178 French children differing in age (4, 5 and 6-year-old) and gender (boys, girls). Results showed age and gender differences in the children's ratings. With increase of age, the stimuli rated as positive decreased, and the stimuli rated as neutral increased. We found gender related differences in the valence ratings, 5 and 6-year-old girls shown more neutral ratings than did boys. We observed a significant difference in ratings between stimulus modalities. Children were more accurate while rating words and pictures than combined words-pictures in each age group and gender. The norms and the set of words and pictures are available in OSF (Center for Open Science), https://osf.io/zcge3/.

A Thematic-Role-Based Approach for Word Sense Disambiguation

The present paper investigates the thematic roles that can be developed for the purpose of Word Sense Disambiguation. In MT systems and electronic word databases, thematic role relations are not clearly included among other semantic relations. Identifying thematic roles of predicates helps in disambiguating word senses and hence producing more accurate translation. For instance, the meaning of the verb ‘eat’ differs depending on the thematic roles it assigns for its Subject and Object. When it assigns an animate Agent for its Subject and food Patient for its Object, it means ‘take in solid food’. However, when it assigns Force for its Subject and metal Theme for its Object, it means ‘cause to deteriorate due to the action of water, air or an acid’. Accordingly, different translations are produced in each context. Selectional restrictions are also tackled in the analysis of the sample verbs. The implementation is made on three MT systems: Al Wafi, Sakhr and Google. They all produce incorrect translations of the sample verbs. A suggested translation is proposed for each verb after analyzing its thematic roles and selectional restrictions. In this way, the present paper is significant since it helps in improving the performance of MT systems. The present paper will focus only on a group of English verbs that convey a variety of meanings. It will show a number of problematic cases in translation that occur due to the lack of thematic roles in the core of the system. After developing thematic roles, it is expected that such cases will be disambiguated.

Construction and Practice of English Writing Model Based on Random Matrix

Writing is an indispensable part of English comprehensive ability. It can test learners’ ability in viewpoint content, text organization, language use, and so on. In view of the many problems existing in the English writing of contemporary primary and middle school students, high school students and even college students, this paper proposes an English writing model based on random matrix. This method collects the necessary English vocabulary information through the English word database and combines the M-P law and the density spectrum value of the hybrid matrix to accurately locate students’ writing needs, clarify their writing needs, identify, and construct dynamic English compositions. The random matrix is used to judge and test the English level threshold of different students, so as to get the abnormal use of words when students write. By comparing the predicted data of students’ abnormal vocabulary with the accurate vocabulary data, different students’ writing ability is judged, and corresponding suggestions are given, and qualified writing formats are provided. The experimental results show that the English writing model based on random matrix has strong learning ability and judgment, and can better achieve one-to-one targeted teaching.

Point‐of‐interest detection from Weibo data for map updating

AbstractPoints‐of‐interest (POIs) geographic information system data are increasingly important for supporting map generation and navigation services, although updating their semantic and location information still largely depends on manual labor. In this study, we propose a novel method to automatically detect the changes in POIs from Chinese text and check‐in position data provided by the Chinese social media platform, Weibo. The proposed method includes three steps: (1) POI name recognition; (2) location confirmation; (3) and change detection. First, we propose recognizing a POI's name from Weibo text using the improved conditional random field algorithm. Then, we detect the location of each named POI by integrating the text address with the check‐in position. The changes in the detected POIs are recognized by extracting the status words from Weibo text and a three‐level status word database. To verify the effectiveness of the proposed method, we examine Wuhan as a case and detect the changes in the commercial POI using real‐world Weibo data collected from January to September 2020. Based on the validation of three common map platforms, the data provided and the manual field investigation of 55 random samples, the identification accuracies for newly added POIs, the unchanged POIs, and expired POIs are approximately 100, 95.8, and 91.7%, respectively.

An English list of trait words including valence, social desirability, and observability ratings.

To enable flexible and controlled research on personality, information processing, and interactions in socio-emotional contexts, the availability of highly controlled stimulus material, especially trait words and related attributes, is indispensable. Existing word databases contain mainly nouns and rating dimensions, and their role in studies within socio-emotional contexts are limited. This study aimed to create an English list of traits (ELoT), a database containing 500 trait adjectives rated by a large sample (n = 822, 57.42% female). The rating categories refer to the perceived valence associated with the traits and their social desirability and observability. Participants of different ages (18 to 65 years of age) and educational levels rated the words in an online survey. Both valence and social desirability ratings showed a bimodal distribution, indicating that most traits were rated either positive (respectively socially desirable) or negative (respectively socially undesirable), with fewer words rated as neutral. For observability, a bell-shaped distribution was found. Results indicated a strong association between valence and social desirability, whereas observability ratings were only moderately associated with the other ratings. Valence and social desirability ratings were not related to participants' age or gender, but observability ratings were different for females and males, and for younger, middle-aged, and older participants. The ELoT is an extensive, freely available database of trait norms. The large sample and the balanced age and gender distributions allow to account for age- and gender-specific effects during stimulus selection.

Judging food by its description: a text mining approach examining the most influential words on restaurant menus

ABSTRACT Previous research regarding menu writing focused on the customers’ perspective, rather than the current practices approach of this study. With the purpose of quantifying and categorizing the word choices most commonly used by restaurant operators, this research used text-mining techniques to develop a database of words used in menu descriptions from a broad cross-section of American casual dining restaurants. The words were categorized and sorted by frequency in order to determine the most common words and categories used. Results showed that categorically, culinary words were used the most in menus while sensory and adjective words were used the least. Within the categories of ingredients, vegetables were most frequently observed while fruits appeared the least in menus. Empirically, the major menu word categories identified in this study can help to update restaurateurs with the current menu word classification. In addition, the words within each category reflect current industry practices, which may be useful to operators in describing menu dishes.

Holistic Persian Handwritten Word Recognition Using Convolutional Neural Network

Due to the cursive-ness and high variability of Persian scripts, the segmentation of handwritten words into sub-words is still a challenging task. These issues could be addressed in a holistic approach by sidestepping segmentation at the character level. In this paper, an end-to-end holistic method based on deep convolutional neural network is proposed to recognize off-line Persian handwritten words. The proposed model uses only five convolutional layers and two fully connected layers for classifying word images effectively, which can lead to a substantial reduction in parameters. The effect of various pooling strategies is also investigated in this paper. The primary goal of this article is to ignore handcrafted feature extraction and attain a generalized and stable word recognition system. The presented model is assessed using two famous handwritten Persian word databases called Sadri and IRANSHAHR. The recognition accuracies were obtained at 98.6% and 94.6%, on Sadri and IRANSHAHR datasets respectively, and outperformed the state-of-the-art methods.

Norming studies of European Portuguese words: A literature review

Words are widely used as stimuli in cognitive and linguistic research. As words may vary on various domains (e.g., lexicosemantic and affective), which can influence performance in many ways, it is essential to select them carefully. However, databases of European Portuguese words are still relatively scarce, and their presentation is spread in various sources. The aims of this work were: (1) to provide a review of the databases containing information on lexicosemantic, affective, and free association norming data for European Portuguese words published since the year 2000; (2) to summarize the definitions of the different variables; and (3) to combine in a single database the norming data available at this point. We found 25 databases with 18 containing lexicosemantic and affective norming data and seven containing word free association norms. These databases include words from various categories (e.g., nouns and adjectives), characterized on different variables (e.g., familiarity, imagery, valence, age of acquisition and animacy). This review provides a useful tool for researchers looking for Portuguese word-related resources. A thorough review of the databases, including some details about each one, is presented. The combined database is shared with the scientific community via the Open Science Framework: https://osf.io/9ta3y/.

Upscaling human activity data: A statistical ecology approach.

Big data require new techniques to handle the information they come with. Here we consider four datasets (email communication, Twitter posts, Wikipedia articles and Gutenberg books) and propose a novel statistical framework to predict global statistics from random samples. More precisely, we infer the number of senders, hashtags and words of the whole dataset and how their abundances (i.e. the popularity of a hashtag) change through scales from a small sample of sent emails per sender, posts per hashtag and word occurrences. Our approach is grounded on statistical ecology as we map inference of human activities into the unseen species problem in biodiversity. Our findings may have applications to resource management in emails, collective attention monitoring in Twitter and language learning process in word databases.