Term-document Matrices Research Articles

Def2Vec: you shall know a word by its definition

AbstractDef2Vec introduces a new perspective on building words embeddings by using dictionary definitions. By leveraging term-document matrices derived from dictionary definitions and employing Latent Semantic Analysis (LSA), our method, Def2Vec, yields embeddings characterized by robust performance and adaptability. Through comprehensive evaluations encompassing token classification, sequence classification and semantic similarity, we show empirically how Def2Vec consistently demonstrates competitiveness with established models like Word2Vec, GloVe, and FastText. Notably, our model’s utilization of all the matrices resulting from LSA factorisation facilitates efficient prediction of embeddings for out-of-vocabulary words, given their definition. By effectively integrating the benefits of dictionary definitions with LSA-based embeddings, Def2Vec builds informative semantic representations, all while minimizing data requirements. In this paper, we run different experiment to assess the quality of our embedding model at word level and at sequence level. Our findings contribute to the ongoing evolution of word embedding methodologies by incorporating structured lexical information and enabling efficient embedding prediction.

The analysis of association rules: Latent class analysis

AbstractAssociation rules are used to extract information from transactional databases with a collection of items also called “tokens” or “words.” The aim of association rule analysis is to indicate what and how items go with what items in a set of transactions called “documents.” This approach is used in the analysis of text records, of blogs in social media and of shopping baskets. We present here an approach to analyze documents using latent class analysis (LCA) clustering of document term matrices. A document term matrix (DTM) consists of rows referring to documents and columns corresponding to items. In binary weights, “1” indicates the presence of a term in a document and “0” otherwise. The clustering of similar documents provides stratified data sets used to enhance the interpretability of measures of interest such as lift, odds ratios and relative linkage disequilibrium. The article demonstrates the approach with two case studies. A first example consists of comments recorded in a survey aimed at pet owners. A second, much larger example, is based on online reviews to crocs sandals. Association rules describe combinations of terms in the pet survey and crocs reviews. In Section 3, we compute, for these case studies, association rule measures of interest defined in Section 2. We first introduce the case studies to motivate the methods proposed here. In Section 4, we provide a new approach with an enhanced interpretations of measures such as lift by comparing them across clusters derived from an LCA of the DTM. A key result is the application of clustered data in analyzing observational data. This enhances generalizability and interpretability of findings from text analytics. The article concludes with a discussion in Section 5.

A comparison of latent semantic analysis and correspondence analysis of document-term matrices

Abstract Latent semantic analysis (LSA) and correspondence analysis (CA) are two techniques that use a singular value decomposition for dimensionality reduction. LSA has been extensively used to obtain low-dimensional representations that capture relationships among documents and terms. In this article, we present a theoretical analysis and comparison of the two techniques in the context of document-term matrices. We show that CA has some attractive properties as compared to LSA, for instance that effects of margins, that is, sums of row elements and column elements, arising from differing document lengths and term frequencies are effectively eliminated so that the CA solution is optimally suited to focus on relationships among documents and terms. A unifying framework is proposed that includes both CA and LSA as special cases. We empirically compare CA to various LSA-based methods on text categorization in English and authorship attribution on historical Dutch texts and find that CA performs significantly better. We also apply CA to a long-standing question regarding the authorship of the Dutch national anthem Wilhelmus and provide further support that it can be attributed to the author Datheen, among several contenders.

Large-Scale Evaluation of Topic Models and Dimensionality Reduction Methods for 2D Text Spatialization.

Topic models are a class of unsupervised learning algorithms for detecting the semantic structure within a text corpus. Together with a subsequent dimensionality reduction algorithm, topic models can be used for deriving spatializations for text corpora as two-dimensional scatter plots, reflecting semantic similarity between the documents and supporting corpus analysis. Although the choice of the topic model, the dimensionality reduction, and their underlying hyperparameters significantly impact the resulting layout, it is unknown which particular combinations result in high-quality layouts with respect to accuracy and perception metrics. To investigate the effectiveness of topic models and dimensionality reduction methods for the spatialization of corpora as two-dimensional scatter plots (or basis for landscape-type visualizations), we present a large-scale, benchmark-based computational evaluation. Our evaluation consists of (1) a set of corpora, (2) a set of layout algorithms that are combinations of topic models and dimensionality reductions, and (3) quality metrics for quantifying the resulting layout. The corpora are given as document-term matrices, and each document is assigned to a thematic class. The chosen metrics quantify the preservation of local and global properties and the perceptual effectiveness of the two-dimensional scatter plots. By evaluating the benchmark on a computing cluster, we derived a multivariate dataset with over 45 000 individual layouts and corresponding quality metrics. Based on the results, we propose guidelines for the effective design of text spatializations that are based on topic models and dimensionality reductions. As a main result, we show that interpretable topic models are beneficial for capturing the structure of text corpora. We furthermore recommend the use of t-SNE as a subsequent dimensionality reduction.

Evaluating criminal justice reform during COVID-19: The need for a novel sentiment analysis package.

The health and safety of incarcerated persons and correctional personnel have been prominent in the U.S. news media discourse during the COVID-19 pandemic. Examining changing attitudes toward the health of the incarcerated population is imperative to better assess the extent to which the general public favors criminal justice reform. However, existing natural language processing lexicons that underlie current sentiment analysis (SA) algorithms may not perform adequately on news articles related to criminal justice due to contextual complexities. News discourse during the pandemic has highlighted the need for a novel SA lexicon and algorithm (i.e., an SA package) tailored for examining public health policy in the context of the criminal justice system. We analyzed the performance of existing SA packages on a corpus of news articles at the intersection of COVID-19 and criminal justice collected from state-level outlets between January and May 2020. Our results demonstrated that sentence sentiment scores provided by three popular SA packages can differ considerably from manually-curated ratings. This dissimilarity was especially pronounced when the text was more polarized, whether negatively or positively. A randomly selected set of 1,000 manually scored sentences, and the corresponding binary document term matrices, were used to train two new sentiment prediction algorithms (i.e., linear regression and random forest regression) to verify the performance of the manually-curated ratings. By better accounting for the unique context in which incarceration-related terminologies are used in news media, both of our proposed models outperformed all existing SA packages considered for comparison. Our findings suggest that there is a need to develop a novel lexicon, and potentially an accompanying algorithm, for analysis of text related to public health within the criminal justice system, as well as criminal justice more broadly.

Requirements for Project Managers—What Do Job Advertisements Say?

The growing number of projects and the key role of project managers in their implementation makes the competencies of managers a subject of many studies. An attempt can be made to determine the project manager competencies that employers appreciate the most through analyses of job advertisements. Due to a very large number of job advertisements, it may be difficult or even impossible to analyze their content manually. A solution may be to fetch and process job advertisements automatically. The main purpose of this paper was to identify the project manager competencies that are the most desired by employers. An analysis of job advertisements was performed to identify the project manager competencies required by employers. Job advertisements were automatically downloaded from online job boards. Fragments of job advertisements that described requirements were analyzed with text mining. The analysis included preprocessing, building of corpora of documents, construction of document-term matrices, application of traditional data mining methods, and Latent Dirichlet Allocation (LDA), which is a popular topic modeling algorithm. After the initial text processing (all characters except letters were removed, uppercase letters were converted to lowercase letters, words deemed useless were removed, and words were converted to their basic form), n-grams were built, and topics identified with LDA were generated. The most frequently used words and n-grams, along with the identified topics, were graphically represented. The meanings of the words and sentences were not analyzed in the text mining analysis of the job advertisements. The analysis did not take into account whether the words appeared side by side in the document-except for the intentional creation of n-grams (such as “communication skill”). The analysis, however, facilitated the identification of certain patterns and regularities in the occurrence of specific strings in the documents (fragments of advertisements describing the requirements). The interpretation of the results is based on the frequency of words and n-grams and frequency of words in topics identified by the LDA algorithm. This paper contributes to science by showing that text mining of job offers can, to some extent, help determine project manager competencies in demand. The method can be used by organizations training future project managers to modify and better adapt curricula to the needs of the labor market. It can be used to monitor the current trends in project manager requirements as well.

Analytical Comparison of Clustering Techniques for the Recognition of Communication Patterns

The systematic processing of unstructured communication data as well as the milestone of pattern recognition in order to determine communication groups in negotiations bears many challenges in Machine Learning. In particular, the so-called curse of dimensionality makes the pattern recognition process demanding and requires further research in the negotiation environment. In this paper, various selected renowned clustering approaches are evaluated with regard to their pattern recognition potential based on high-dimensional negotiation communication data. A research approach is presented to evaluate the application potential of selected methods via a holistic framework including three main evaluation milestones: the determination of optimal number of clusters, the main clustering application, and the performance evaluation. Hence, quantified Term Document Matrices are initially pre-processed and afterwards used as underlying databases to investigate the pattern recognition potential of clustering techniques by considering the information regarding the optimal number of clusters and by measuring the respective internal as well as external performances. The overall research results show that certain cluster separations are recommended by internal and external performance measures by means of a holistic evaluation approach, whereas three of the clustering separations are eliminated based on the evaluation results.

Importance of data preparation when analysing written responses to open-ended questions: An empirical assessment and comparison with manual coding

In a world where consumer texts grow more numerous each day, automated text analysis can deliver valuable insights about consumer attitudes and behaviours. The present research was methodological in nature and focused on pre-processing of text data, which generally is the most time-consuming stage of analysis. Using responses to an open-ended question from 4341 consumers, document-term matrices (DTM) were created from varying combinations of n-grams (unigrams, bigrams, trigrams and combinations hereof), stemming (yes or no) and low-frequency term thresholding (retaining all terms or excluding those used < 0.1%, <1% or < 5%). By comparison to a fixed standard – manually derived content coded of respondents’ answers – the relative impact of the three pre-processing steps were assessed. PLS-DA was used to do so, and classifier performance was evaluated using AUC-ROC scores. Inclusion of bigrams and trigrams in DTMs did not influence classification performance and stemming had only a minor impact. Inclusion of all and very rare features (<0.1%) improved classification performance. The results were invariant of sample size and replicated in subsets of 2000, 1000 and 500 participants. The results may be specific to the short length of the answers (median words = 4), although they held in a sub-sample of the 500 longest answers (median words = 41). Future research should directly test the influence of these pre-processing steps, for example, through topic modelling.

Evaluating Criminal Justice Reform During COVID-19: The Need for a Novel Sentiment Analysis Package

Existing natural language processing lexicons that underlie current sentiment analysis (SA) algorithms may not perform adequately in certain academic disciplines depending on contextual complexities. The health and safety of incarcerated persons and correctional personnel have been prominent in news media discourse during the COVID-19 pandemic, potentially highlighting the need for a novel SA lexicon and algorithm that is tailored for the examination of public health policy in the context of the criminal justice system. We utilized a text corpus consisting of news articles at the intersection of COVID-19 and criminal justice to analyze the performance of existing lexicons collected across state-level outlets between January and May 2020. Our results demonstrated that sentence sentiment scores provided by three popular SA packages differ considerably from manually-curated ratings. This dissimilarity was especially pronounced when the text was more polarized, whether negatively or positively. A randomly selected set of 1,000 manually scored sentences, and the corresponding binary document term matrices, were used to train two new sentiment prediction algorithms (i.e., linear regression and random forest regression) to verify the performance of the manually-curated ratings. By better accounting for the unique context in which incarceration-related terminologies are used in news media, both of our proposed models outperformed all existing SA packages considered for comparison. Our findings suggest that there is a need to develop a novel lexicon, and potentially an accompanying algorithm, for analysis of text related to public health within the criminal justice system, as well as criminal justice more broadly.

Textual data summarization using the Self-Organized Co-Clustering model

Recently, different studies have demonstrated the use of co-clustering, a data mining technique which simultaneously produces row-clusters of observations and column-clusters of features. The present work introduces a novel co-clustering model to easily summarize textual data in a document-term format. In addition to highlighting homogeneous co-clusters as other existing algorithms do we also distinguish noisy co-clusters from significant co-clusters, which is particularly useful for sparse document-term matrices. Furthermore, our model proposes a structure among the significant co-clusters, thus providing improved interpretability to users. The approach proposed contends with state-of-the-art methods for document and term clustering and offers user-friendly results. The model relies on the Poisson distribution and on a constrained version of the Latent Block Model, which is a probabilistic approach for co-clustering. A Stochastic Expectation-Maximization algorithm is proposed to run the model’s inference as well as a model selection criterion to choose the number of co-clusters. Both simulated and real data sets illustrate the efficiency of this model by its ability to easily identify relevant co-clusters.

Can topic models be used in research evaluations? Reproducibility, validity, and reliability when compared with semantic maps

AbstractWe replicate and analyze the topic model which was commissioned to King’s College and Digital Science for the Research Evaluation Framework (REF 2014) in the United Kingdom: 6,638 case descriptions of societal impact were submitted by 154 higher-education institutes. We compare the Latent Dirichlet Allocation (LDA) model with Principal Component Analysis (PCA) of document-term matrices using the same data. Since topic models are almost by definition applied to text corpora which are too large to read, validation of the results of these models is hardly possible; furthermore the models are irreproducible for a number of reasons. However, removing a small fraction of the documents from the sample—a test for reliability—has on average a larger impact in terms of decay on LDA than on PCA-based models. The semantic coherence of LDA models outperforms PCA-based models. In our opinion, results of the topic models are statistical and should not be used for grant selections and micro decision-making about research without follow-up using domain-specific semantic maps.

The analysis of source code plagiarism in basic programming course

The purpose of this research is to recognize the types of plagiarism that occurred in basic programming courses as a preliminary research in developing an intelligent system to detect source plagiarism in programming class. This research use a hierarchical clustering method which is implemented in term-document matrices. In this research, we propose the use of keywords in the programming language informing the term-document matrix. The result is normalized and then we use Euclidean distance to normalize data for similarity measurement. The result shows that this method is quite effective to classify objects of data indicated by plagiarism. Some types of plagiarism which are mostly occurred are formatting source code, changing comment, renaming identifier, and adding statements or redundant variables.

Orthogonal nonnegative matrix tri-factorization based on Tweedie distributions

Orthogonal nonnegative matrix tri-factorization (ONMTF) is a biclustering method using a given nonnegative data matrix and has been applied to document-term clustering, collaborative filtering, and so on. In previously proposed ONMTF methods, it is assumed that the error distribution is normal. However, the assumption of normal distribution is not always appropriate for nonnegative data. In this paper, we propose three new ONMTF methods, which respectively employ the following error distributions: normal, Poisson, and compound Poisson. To develop the new methods, we adopt a k-means based algorithm but not a multiplicative updating algorithm, which was the main method used for obtaining estimators in previous methods. A simulation study and an application involving document-term matrices demonstrate that our method can outperform previous methods, in terms of the goodness of clustering and in the estimation of the factor matrix.

No Longer Lost in Translation: Evidence that Google Translate Works for Comparative Bag-of-Words Text Applications

Automated text analysis allows researchers to analyze large quantities of text. Yet, comparative researchers are presented with a big challenge: across countries people speak different languages. To address this issue, some analysts have suggested using Google Translate to convert all texts into English before starting the analysis (Lucas et al. 2015). But in doing so, do we get lost in translation? This paper evaluates the usefulness of machine translation for bag-of-words models—such as topic models. We use the europarl dataset and compare term-document matrices (TDMs) as well as topic model results from gold standard translated text and machine-translated text. We evaluate results at both the document and the corpus level. We first find TDMs for both text corpora to be highly similar, with minor differences across languages. What is more, we find considerable overlap in the set of features generated from human-translated and machine-translated texts. With regard to LDA topic models, we find topical prevalence and topical content to be highly similar with again only small differences across languages. We conclude that Google Translate is a useful tool for comparative researchers when using bag-of-words text models.

Directional co-clustering

Co-clustering addresses the problem of simultaneous clustering of both dimensions of a data matrix. When dealing with high dimensional sparse data, co-clustering turns out to be more beneficial than one-sided clustering even if one is interested in clustering along one dimension only. Aside from being high dimensional and sparse, some datasets, such as document-term matrices, exhibit directional characteristics, and the $$L_2$$ normalization of such data, so that it lies on the surface of a unit hypersphere, is useful. Popular co-clustering assumptions such as Gaussian or Multinomial are inadequate for this type of data. In this paper, we extend the scope of co-clustering to directional data. We present Diagonal Block Mixture of Von Mises–Fisher distributions (dbmovMFs), a co-clustering model which is well suited for directional data lying on a unit hypersphere. By setting the estimate of the model parameters under the maximum likelihood (ML) and classification ML approaches, we develop a class of EM algorithms for estimating dbmovMFs from data. Extensive experiments, on several real-world datasets, confirm the advantage of our approach and demonstrate the effectiveness of our algorithms.

Model-based co-clustering for the effective handling of sparse data

With the exponential growth of text documents on the web, there is a genuine need for techniques that organize terms and documents, simultaneously, into meaningful clusters, thereby making large datasets easier to handle and interpret. Several block diagonal clustering algorithms have proven successful in identifying co-clusters of documents and words. However, despite their effectiveness, most of the existing methods do not provide a parameterizable model for tackling the problem of block diagonal identification. In this paper, we rely on mixture models, which offer strong theoretical foundations and considerable flexibility. More precisely, we propose a parsimonious latent block model based on the mixture of Poisson distributions and tailored for sparse high dimensional data such as document-term matrices. In order to efficiently estimate the model parameters, we derive two scalable co-clustering algorithms based on variational inference. Empirical results obtained on several real-world text datasets highlight the advantages of the proposed model and the corresponding co-clustering algorithms.

Sparse Poisson Latent Block Model for Document Clustering

Over the last decades, several studies have demonstrated the importance of co-clustering to simultaneously produce groups of objects and features. Even to obtain object clusters only, using co-clustering is often more effective than one-way clustering, especially when considering sparse high dimensional data. In this paper, we present a novel generative mixture model for co-clustering such data. This model, the Sparse Poisson Latent Block Model (SPLBM), is based on the Poisson distribution, which arises naturally for contingency tables, such as document-term matrices. The advantages of SPLBM are two-fold. First, it is a rigorous statistical model which is also very parsimonious. Second, it has been designed from the ground up to deal with data sparsity problems. As a consequence, in addition to seeking homogeneous blocks, as other available algorithms, it also filters out homogeneous but noisy ones due to the sparsity of the data. Experiments on various datasets of different size and structure show that an algorithm based on SPLBM clearly outperforms state-of-the-art algorithms. Most notably, the SPLBM-based algorithm presented here succeeds in retrieving the natural cluster structure of difficult, unbalanced datasets which other known algorithms are unable to handle effectively.

Topic modelling with morphologically analyzed vocabularies

Probabilistic topic modeling is a text mining technique that allows to extract sets of term probability distributions which can intuitively be interpreted as latent topics. The extraction in most techniques uses only document term frequency matrices as input data. Moreover, topic models estimate posterior document-topic distributions useful for intelligent document retrieval query processing. This paper discusses two approaches to topic modeling involving Dirichlet distributions and Dirichlet processes. However, these and related approaches presume suitable text preprocessing in order to keep parameter spaces for estimations from training text corpora at manageable sizes. In the present paper, we discuss the influence of morphological preprocessing of training texts. Morphological analysis is a computer linguistic discipline that allows to decompose observed terms into base lemmata. This is effected by a deep analysis of the observed terms as opposed to straightforward prefix or postfix elimination used in conventional stemming algorithms. Morphological preprocessing is especially effective in inflection rich languages like, e.g. Finnish or German, and effectively reduces the training vocabulary size. In addition, morphological preprocessing allows for decomposing compound words. It is of considerable interest to study the influence of morphological preprocessing on text mining and statistical topic models. In experiments reported in the application section of this paper, significant changes of the frequency structure of document term matrices were found. Interestingly, these changes also led to substantial improvements in model quality indicators of topic models due to morphological preprocessing. Steps for further research are suggested in the concluding section.

Graph modularity maximization as an effective method for co-clustering text data

In this paper we show how the modularity measure can serve as a useful criterion for co-clustering document-term matrices. We present and investigate the performance of CoClus, a novel, effective block-diagonal co-clustering algorithm which directly maximizes this modularity measure. The maximization is performed using an iterative alternating optimization procedure, in contrast to algorithms that use spectral relaxations of the discrete optimization problems. Extensive comparative experiments performed on various document-term datasets demonstrate that this approach is very effective, stable, and outperforms other block-diagonal co-clustering algorithms devoted to the same task. Another important advantage of using modularity in the co-clustering context is that it provides a novel, simple way of determining the appropriate number of co-clusters.Availability: an implementation of CoClus is available as part of the recently released coclust Python package which is available at: https://pypi.python.org/pypi/coclust

Human Rights Texts: Converting Human Rights Primary Source Documents into Data.

We introduce and make publicly available a large corpus of digitized primary source human rights documents which are published annually by monitoring agencies that include Amnesty International, Human Rights Watch, the Lawyers Committee for Human Rights, and the United States Department of State. In addition to the digitized text, we also make available and describe document-term matrices, which are datasets that systematically organize the word counts from each unique document by each unique term within the corpus of human rights documents. To contextualize the importance of this corpus, we describe the development of coding procedures in the human rights community and several existing categorical indicators that have been created by human coding of the human rights documents contained in the corpus. We then discuss how the new human rights corpus and the existing human rights datasets can be used with a variety of statistical analyses and machine learning algorithms to help scholars understand how human rights practices and reporting have evolved over time. We close with a discussion of our plans for dataset maintenance, updating, and availability.