Bag-of-words Research Articles

Semantic Similarity Comparison Between Production Line Failures for Predictive Maintenance

With the introduction of Industry 4.0 into our lives and the creation of smart factories, predictive maintenance has become even more important. Predictive maintenance systems are often used in the manufacturing industry. On the other hand, text analysis and Natural Language Processing (NLP) techniques are gaining a lot of attention by both research and industry due to their ability to combine natural languages and industrial solutions. There is a great increase in the number of studies on NLP in the literature. Even though there are studies in the field of NLP in predictive maintenance systems, no studies were found on Turkish NLP for predictive maintenance. This study focuses on the similarity analysis of failure texts that can be used in the predictive maintenance system we developed for VESTEL, one of the leading consumer electronics manufacturers in Turkey. In the manufacturing industry, operators record descriptions of failure that occur on production lines as short texts. However, these descriptions are not often used in predictive maintenance work. In this study, semantic text similarities between fault definitions in the production line were compared using traditional word representations, modern word representations and Transformer models. Levenshtein, Jaccard, Pearson, and Cosine scales were used as similarity measures and the effectiveness of these measures were compared. Experimental data including failure texts were obtained from a consumer electronics manufacturer in Turkey. When the experimental results are examined, it is seen that the Jaccard similarity metric is not successful in grouping semantic similarities according to the other three similarity measures. In addition, Multilingual Universal Sentence Encoder (MUSE), Language-agnostic BERT Sentence Embedding (LAbSE), Bag of Words (BoW) and Term Frequency - Inverse Document Frequency (TF-IDF) outperform FastText and Language-Agnostic Sentence Representations (LASER) models in semantic discovery of error identification in embedding methods. Briefly to conclude, Pearson and Cosine are more effective at finding similar failure texts; MUSE, LAbSE, BoW and TF-IDF methods are more successful at representing the failure text.

A sophisticated semantic analysis framework using an intelligent tweet data clustering and classification methodologies

Developing an efficient twitter sentiment analysis framework is one of the most challenging and demanding chores in current days. In the conventional works, various machine/deep learning based detection mechanisms are developed for tweet analysis. It encompasses the methodologies of data clustering, optimization, and classification, but it has the major problems of difficult to understand, complexity in mathematical computations, high time consumption, and error rate. Therefore, the proposed research work intends to implement an intelligent and advanced data mining techniques to design and develop a sentiment analysis framework. At first, the preprocessing is performed to normalize the dataset for generating the noise free balanced dataset. After that, the Hierarchical Tweet Expression Clustering (HTEC) algorithm is used to cluster the attributes according to the distance for simplifying the process of classification. Moreover, the multi-faceted features such as vector space Bag-of-Words (BOW), and Term Frequency – Inverse Document Frequency (IDM) are extracted for classifier training and testing operations. Finally, the Spatial Dense Bi-directional Long Short Memory (SDBi-LSTM) classification methodology is used for predicting the sentiment as positive, negative, or neutral. For evaluation, the Stanford Twitter Sentiment Test Set (STS-Test/Sentiment140), GOP debate and IMDB datasets are used to validate the results of the proposed system by using the parameters of accuracy, f1-score, training time, testing time, true positive rate, false positive rate, and etc. Also, the obtained results are compared with the recent prediction models for proving the superiority of the proposed framework.

HFNet-SLAM: An Accurate and Real-Time Monocular SLAM System with Deep Features.

Image tracking and retrieval strategies are of vital importance in visual Simultaneous Localization and Mapping (SLAM) systems. For most state-of-the-art systems, hand-crafted features and bag-of-words (BoW) algorithms are the common solutions. Recent research reports the vulnerability of these traditional algorithms in complex environments. To replace these methods, this work proposes HFNet-SLAM, an accurate and real-time monocular SLAM system built on the ORB-SLAM3 framework incorporated with deep convolutional neural networks (CNNs). This work provides a pipeline of feature extraction, keypoint matching, and loop detection fully based on features from CNNs. The performance of this system has been validated on public datasets against other state-of-the-art algorithms. The results reveal that the HFNet-SLAM achieves the lowest errors among systems available in the literature. Notably, the HFNet-SLAM obtains an average accuracy of 2.8 cm in EuRoC dataset in pure visual configuration. Besides, it doubles the accuracy in medium and large environments in TUM-VI dataset compared with ORB-SLAM3. Furthermore, with the optimisation of TensorRT technology, the entire system can run in real-time at 50 FPS.

Hybrid classifier model with tuned weights for human activity recognition

A wide variety of uses, such as video interpretation and surveillance, human-robot interaction, healthcare, and sport analysis, among others, make this technology extremely useful, human activity recognition has received a lot of attention in recent decades. human activity recognition from video frames or still images is a challenging procedure because of factors including viewpoint, partial occlusion, lighting, background clutter, scale differences, and look. Numerous applications, including human-computer interfaces, robotics for the analysis of human behavior, and video surveillance systems all require the activity recognition system. This work introduces the human activity recognition system, which includes 3 stages: preprocessing, feature extraction, and classification. The input video (image frames) are subjected for preprocessing stage which is processed with median filtering and background subtraction. Several features, including the Improved Bag of Visual Words, the local texton XOR pattern, and the Spider Local Picture Feature (SLIF) based features, are extracted from the pre-processed image. The next step involves classifying data using a hybrid classifier that blends Bidirectional Gated Recurrent (Bi-GRU) and Long Short Term Memory (LSTM). To boost the effectiveness of the suggested system, the weights of the Long Short Term Memory (LSTM) and Bidirectional Gated Recurrent (Bi-GRU) are both ideally determined using the Improved Aquila Optimization with City Block Distance Evaluation (IACBD) method. Finally, the effectiveness of the suggested approach is evaluated in comparison to other traditional models using various performance metrics.

Data reduction for X-ray serial crystallography using machine learning.

Serial crystallography experiments produce massive amounts of experimental data. Yet in spite of these large-scale data sets, only a small percentage of the data are useful for downstream analysis. Thus, it is essential to differentiate reliably between acceptable data (hits) and unacceptable data (misses). To this end, a novel pipeline is proposed to categorize the data, which extracts features from the images, summarizes these features with the 'bag of visual words' method and then classifies the images using machine learning. In addition, a novel study of various feature extractors and machine learning classifiers is presented, with the aim of finding the best feature extractor and machine learning classifier for serial crystallography data. The study reveals that the oriented FAST and rotated BRIEF (ORB) feature extractor with a multilayer perceptron classifier gives the best results. Finally, the ORB feature extractor with multilayer perceptron is evaluated on various data sets including both synthetic and experimental data, demonstrating superior performance compared with other feature extractors and classifiers.

Automated Identification of Disaster News for Crisis Management using Machine Learning

A lot of news sources picked up on Typhoon Rai (also known locally as Typhoon Odette), along with fake news outlets. The study honed in on the issue, to create a model that can identify between legitimate and illegitimate news articles. With this in mind, we chose the following machine learning algorithms in our development: Logistic Regression, Random Forest and Multinomial Naive Bayes. Bag of Words, TF-IDF and Lemmatization were implemented in the Model. Gathering 160 datasets from legitimate and illegitimate sources, the machine learning was trained and tested. By combining all the machine learning techniques, the Combined BOW model was able to reach an accuracy of 91.07%, precision of 88.33%, recall of 94.64%, and F1 score of 91.38% and Combined TF-IDF model was able to reach an accuracy of 91.18%, precision of 86.89%, recall of 94.64%, and F1 score of 90.60%.

Content-Based Image Retrieval System using Color Moment and Bag of Visual Words with Local Binary Pattern

Researchers have recently focused their attention on Content-Based Image Retrieval (CBIR). It has emerged as one of the most fascinating areas in image processing and computer vision. With CBIR, the most comparable pictures that match the query image are pulled from an image database. As a result, it necessitates feature extraction (Local / Global) and similarity calculation. This paper uses a CBIR technique to determine the images that best match the image query by utilizing both global and local image features. A color moment is used for global features to describe the complete image. Local Binary Pattern (LBP) as a local feature, on the other hand, extracts interest points by building a Bag of Visual Words (BoVW). The distance between the query and database image features is computed using the Euclidean distance. Precision and recall are computed on the Corel-1K dataset to assess the retrieval performance.

Mining culture from professional discourse: a lexicon-based hybrid method

Professional discourse is a culturally situated construct. However, due to methodological constrains, culture as a discourse determinant revealed by analytical traditions cannot easily be generalized. By integrating culture representative word stems generated by a bag-of-word (BoW) model into a latent Dirichlet allocation (LDA) model, this paper explored a lexicon-based hybrid approach to mining corporate cultures from MD&As. Two separate corpora, the corporation culture corpus sized 801,219 tokens and the MD&A corpus with 2,199,109 tokens, were used. Key word stems drawn from the corporate culture corpus by the BoW showed a strong capacity of distinguishing implicit corporate values between texts, while the LDA model applied to the MD&A corpus succeeded in identifying three types of cultures in MD&As. Case studies confirmed the credibility of the classification. The hybrid method extends the literature by providing a quantitative tool for the qualitative study of culture in professional discourse, and shows implications for what interdiscursivity in MD&As looks like and how it works to formulate a renewable cycle in discourse realization.

Proposed spatio‐temporal features for human activity classification using ensemble classification model

SummaryClassifying human actions from still images or video sequences is a demanding task owing to issues, like lighting, backdrop clutter, variations in scale, partial occlusion, viewpoint, and appearance. A lot of appliances, together with video systems, human–computer interfaces, and surveillance necessitate a compound action recognition system. Here, the proposed system develops a novel scheme for HAR. Initially, filtering as well as background subtraction is done during preprocessing. Then, the features including local binary pattern (LBP), bag of the virtual word (BOW), and the proposed local spatio‐temporal features are extracted. Then, in the recognition phase, an ensemble classification model is introduced that includes Recurrent Neural networks (RNN 1 and RNN 2) and Multi‐Layer Perceptron (MLP 1 and MLP 2). The features are classified using RNN 1 and RNN 2, and the outputs from RNN 1 and RNN 2 are further classified using MLP 1 and MLP 2, respectively. Finally, the outputs attained from MLP 1 and MLP 2 are averaged and the final classified output is obtained. At last, the superiority of the developed approach is proved on varied measures.

MBi-GRUMCONV: A novel Multi Bi-GRU and Multi CNN-Based deep learning model for social media sentiment analysis

Today, internet and social media is used by many people, both for communication and for expressing opinions about various topics in many domains of life. Various artificial intelligence technologies-based approaches on analysis of these opinions have emerged natural language processing in the name of different tasks. One of these tasks is Sentiment analysis, which is a popular method aiming the task of analyzing people’s opinions which provides a powerful tool in making decisions for people, companies, governments, and researchers. It is desired to investigate the effect of using multi-layered and different neural networks together on the performance of the model to be developed in the sentiment analysis task. In this study, a new, deep learning-based model was proposed for sentiment analysis on IMDB movie reviews dataset. This model performs sentiment classification on vectorized reviews using two methods of Word2Vec, namely, the Skip Gram and Continuous Bag of Words, in three different vector sizes (100, 200, 300), with the help of 6 Bidirectional Gated Recurrent Units and 2 Convolution layers (MBi-GRUMCONV). In the experiments conducted with the proposed model, the dataset was split into 80%-20% and 70%-30% training-test sets, and 10% of the training splits were used for validation purposes. Accuracy and F1 score criteria were used to evaluate the classification performance. The 95.34% accuracy of the proposed model has outperformed the studies in the literature. As a result of the experiments, it was found that Skip Gram has a better contribution to classification success.

An Emotion-Based Rating System for Books Using Sentiment Analysis and Machine Learning in the Cloud

Sentiment analysis (SA), and emotion detection and recognition from text (EDRT) are recent areas of study that are closely related to each other. Sentiment analysis strives to identify and detect neutral, positive, or negative feelings from text. On the other hand, emotion analysis seeks to identify and distinguish types of feelings such as happiness, surprise, grief, disgust, fear, and anger through the expression of texts. We suggest a four-level strategy in this paper for recommending the best book to users. The levels include semantic network grouping of comparable sentences, sentiment analysis, reviewer clustering, and recommendation system. The semantic network groups comparable sentences at the first level utilizing pre-processed data from reviewer and book datasets using the parts of speech (POS) tagger. In order to extract keywords from the pre-processed data, feature extraction uses the bag of words (BOW) and term frequency-inverse document frequency (TF-IDF) approaches. SA is performed at the second level in two phases: training and testing, employing deep learning methodologies such as convolutional neural networks (CNN)-long short-term memory (LSTM). The results of this level are sent into the third level (clustering), which uses the clustering method to group the reviewers by age, location, and gender. In the last level, the model assessment is carried out with accuracy, precision, recall, sensitivity, specificity, G-mean, and F1-measure. The book suggestion system is designed to provide the highest level of accuracy within a minimum number of epochs when compared to the state-of-the methods, SVM, CNN, ANN, LSTM, and Bi-directional (BI)-LSTM.

Predicting bankruptcy of firms using earnings call data and transfer learning.

Business collapse is a common event in economies, small and big alike. A firm's health is crucial to its stakeholders like creditors, investors, partners, etc. and prediction of the upcoming financial crisis is significantly important to devise appropriate strategies to avoid business collapses. Bankruptcy prediction has been regarded as a critical topic in the world of accounting and finance. Methodologies and strategies have been investigated in the research domain for predicting company bankruptcy more promptly and accurately. Conventionally, predicting the financial risk and bankruptcy has been solely achieved using the historic financial data. CEOs also communicate verbally via press releases and voice characteristics, such as emotion and tone may reflect a company's success, according to anecdotal evidence. Companies' publicly available earning calls data is one of the main sources of information to understand how businesses are doing and what are expectations for the next quarters. An earnings call is a conference call between the management of a company and the media. During the call, management offers an overview of recent performance and provides a guide for the next quarter's expectations. The earning calls summary provided by the management can extract CEO's emotions using sentiment analysis. This article investigates the prediction of firms' health in terms of bankruptcy and non-bankruptcy based on emotions extracted from earning calls and proposes a deep learning model in this regard. Features extracted from long short-term memory (LSTM) network are used to train machine learning models. Results show that the models provide results with a high score of 0.93, each for accuracy and F1 when trained on LSTM extracted feature from synthetic minority oversampling technique (SMOTE) balanced data. LSTM features provide better performance than traditional bag of words and TF-IDF features.

Hoax COVID-19 News Detection Based on Sentiment Analysis in Indonesian using Support Vector Machine (SVM) Method

The increasing use of technology makes it easier for information media such as news to be disseminated and does not demand possibilities, there is a lot of hoax news spreading. Twitter is one of the media most frequently used by the public to access and disseminate information. This research will focus on detecting Indonesian language COVID-19 news taken from Twitter. Detection of hoax news can be assisted by using sentiment analysis, one of the uses of classification text. Support Vector Machine (SVM) can be used to perform sentiment analysis tasks. After getting the sentiment analysis results, the hoax detection process will use the Bag of Words. Bag of Words is a collection of word dictionaries for weighting words to determine specific labels. The built SVM model succeeded in classifying tweet repliessentiment with an average accuracy of 83.17% with a threshold of 35%. At the same time, the hoax detection process gets the best accuracy of 62.5% with a threshold of -5 or -6.

Natural language processing of admission notes to predict postpartum hemorrhage

Artificial intelligence is being integrated into practice to support clinical decision-making. The objective of this work was to determine if natural language processing (NLP) of admission notes can predict postpartum hemorrhage (PPH) during the delivery admission, a common obstetric complication leading to severe maternal morbidity. This is a retrospective study of individuals admitted for delivery between July 2016 and December 2020 at a single academic hospital. The outcome of interest was postpartum hemorrhage, defined as estimated blood loss ≥1000 mL. Admission notes were prepared for text processing by excluding stop words, punctuation, terms that occurred very frequently ( >70%) and infrequently (< 1%). The ‘bag of words’ approach with a LASSO model was used to determine if notes were predictive of PPH. Single word vocabularies were trained in 75% of the data and model performance examined in the withheld 25% (test set). Models were then externally validated in a separate hospital within the same health system. Areas under the receiver operating characteristic curve (AUC) were calculated to assess for model discrimination, and calibration plots were generated. The analysis included 12,590 patients in the derivation group and 20,775 patients in the external validation group. The rates of PPH were 8.2% in the test set and 8.7% in the validation set, respectively. AUCs were 0.74 (95% CI 0.71, 0.77) and 0.67 (95% CI 0.66, 0.69) in the test and validation set. The calibration plots revealed a group of patients at significantly higher risk of PPH (those in the top decile of predicted risk compared to others): rates of PPH 27.7% vs 6.2% and 23.0% vs 7.1% in the test and validation sets, respectively (p< 0.001). NLP of EHR documentation generated during routine clinical care identified individuals at highest risk for PPH. Next steps involve comparing predictions to clinician assessment and piloting the model in clinical practice. If successful, this type of tool could ultimately be incorporated to provide real-time information to care teams.View Large Image Figure ViewerDownload Hi-res image Download (PPT)

Drug Usage Safety from Drug Reviews with Hybrid Machine Learning Approach

With the increasing usage of drugs to remedy different diseases, drug safety has become crucial over the past few years. Often medicine from several companies is offered for a single disease that involves the same/similar substances with slightly different formulae. Such diversification is both helpful and dangerous as such medicine proves to be more effective or shows side effects to different patients. Despite clinical trials, side effects are reported when the medicine is used by the mass public, of which several such experiences are shared on social media platforms. A system capable of analyzing such reviews could be very helpful to assist healthcare professionals and companies for evaluating the safety of drugs after it has been marketed. Sentiment analysis of drug reviews has a large potential for providing valuable insights into these cases. Therefore, this study proposes an approach to perform analysis on the drug safety reviews using lexicon-based and deep learning techniques. A dataset acquired from the ‘Drugs.Com’ containing reviews of drug-related side effects and reactions, is used for experiments. A lexicon-based approach, Textblob is used to extract the positive, negative or neutral sentiment from the review text. Review classification is achieved using a novel hybrid deep learning model of convolutional neural networks and long short-term memory (CNN-LSTM) network. The CNN is used at the first level to extract the appropriate features while LSTM is used at the second level. Several well-known machine learning models including logistic regression, random forest, decision tree, and AdaBoost are evaluated using term frequency-inverse document frequency (TF-IDF), a bag of words (BoW), feature union of (TF-IDF + BoW), and lexicon-based methods. Performance analysis with machine learning models, long short term memory and convolutional neural network models, and state-of-the-art approaches indicate that the proposed CNN-LSTM model shows superior performance with an 0.96 accuracy. We also performed a statistical significance T-test to show the significance of the proposed CNN-LSTM model in comparison with other approaches.

Two Stage Job Title Identification System for Online Job Advertisements

Data science techniques are powerful tools for extracting knowledge from large datasets. Analyzing the job market by classifying online job advertisements (ads) has recently received much attention. Various approaches for multi-label classification (e.g., self-supervised learning and clustering) have been developed to identify the occupation from a job advertisement and have achieved a satisfying performance. However, these approaches require labeled datasets with hundreds of thousands of examples and focus on specific databases such as the Occupational Information Network (O*NET) that are more adapted to the US job market. In this paper, we present a two-stage job title identification methodology to address the case of small datasets. We use Bidirectional Encoder Representations from Transformers (BERT) to first classify the job ads according to their corresponding sector (e.g., Information Technology, Agriculture). Then, we use unsupervised machine learning algorithms and some similarity measures to find the closest matching job title from the list of occupations within the predicted sector. We also propose a novel document embedding strategy to address the issues of processing and classifying job ads. Our experimental results show that the proposed two-stage approach improves the job title identification accuracy by 14% to achieve more than 85% in some sectors. Moreover, we found that incorporating document embedding-based approaches such as weighting strategies and noise removal improves the classification accuracy by 23.5% compared to approaches based on the Bag of words model. Further evaluations verify that the proposed methodology either outperforms or performs at least as well as the state-of-the-art methods. Applying the proposed methodology to Moroccan job market data has helped identify emerging and high-demand occupations in Morocco.

Stress detection system for social media users

Stress detection is a growing topic in the field of natural language processing. The study of stress detection for mental health prediction has been proven to benefit the development of recommender systems and automated mental health assessments in previous studies. Additionally, the widespread usage of social media has served as a potential data source for developing such models. Our research tried to detect whether the users of social media were under stress or not. We used a dataset from Dreaddit consisting of posts from one of the popular social media platforms, Reddit. We propose a machine learning model consisting of Support Vector Machine (SVM), Naïve Bayes, Decision Tree, Random Forest, Bag of Words, and Term Frequency – inverse document frequency (TF-IDF) for stress detection. The final evaluation of the model achieved an 80.00% F-1 Score and 75.00% accuracy, and both were scored by SVM.

Object Recognition for Humanoid Robots Using Full Hand Tactile Sensor

With the development of human-like robotic technologies, robots have been advanced to sense and recognize objects, which is one branch of artificial intelligence to mimic human reactions. Although high-performance computer technologies are used, the ability of object recognition by touch is still low due to the lack of proper sensors. Therefore, this research studied the object learning and recognition system through robot touches by developing an artificial sensory system acting as an electronic skin with tactile sensors. The Tactile Sensor is developed in this research, consisting of 15 Tactile Sensor Arrays and the palm’s touchpoints. Furthermore, recognition analysis was developed on Bag of Word (BoW) and Convolution Neural Network (CNN) algorithms. With the BoW technique, using Support Vector Machine (SVM) as a classifier with Moment Analysis Descriptor (MA) provided the highest accuracy, showing more than 80.15% accuracy from five grasping of an object. With the CNN approach, InceptionNetV3 provided the highest accuracy of 98.28% from only one capture of an object.

RadFormer: Transformers with global-local attention for interpretable and accurate Gallbladder Cancer detection.

We propose a novel deep neural network architecture to learn interpretable representation for medical image analysis. Our architecture generates a global attention for region of interest, and then learns bag of words style deep feature embeddings with local attention. The global, and local feature maps are combined using a contemporary transformer architecture for highly accurate Gallbladder Cancer (GBC) detection from Ultrasound (USG) images. Our experiments indicate that the detection accuracy of our model beats even human radiologists, and advocates its use as the second reader for GBC diagnosis. Bag of words embeddings allow our model to be probed for generating interpretable explanations for GBC detection consistent with the ones reported in medical literature. We show that the proposed model not only helps understand decisions of neural network models but also aids in discovery of new visual features relevant to the diagnosis of GBC. Source-code is available at https://github.com/sbasu276/RadFormer.

Workers’ perceptions of mHealth services for physical activity and mental health: A qualitative study using a text-mining method

Objectives: This study explored the perceptions of workers regarding mobile health (mHealth) services for physical activity and mental health. Methods: Single, online, one-on-one, 60-minute semi-structured interviews were conducted with workers living or working in Tokyo, Japan. The transcribed text corpus of the interviews was used as data to explore their perceptions. The top 200 nouns in the utterances were extracted and modeled using a bag of words (BoW) and embedded into a two-dimensional space. Results: The interviews extracted 476 utterances with 1,294 nouns from the 12 workers (7 males and 5 females). A total of 10 themes were estimated from the top 200 nouns. The participants mostly agreed that physical activity was effective in improving their mental health. They needed individualized, attractive, and easy mHealth services. Other estimated themes were: limited effects of non-leisure physical activity on mental health, lower priority on physical activity rather than sleep and rest, reluctance to share the data within the groups, and difficulties in wearing the devices to measure physical activity due to work rules. Conclusion: Workers' perceptions of mHealth services were consistent with previous findings: the need for individualization, attraction, and ease of use. In the working population, mHealth services for physical activity and mental health should consider working conditions and focus on leisure-time physical activity. Social sharing might not be a mandatory technique because of the private topics and variety of associations between physical activity and mental health.