Term-document Matrices Research Articles

Human Rights Texts: Converting Human Rights Primary Source Documents into Data

We introduce and make publicly available a large corpus of digitized primary source human rights documents which are published annually by monitoring agencies that include Amnesty International, Human Rights Watch, the Lawyers Committee for Human Rights, and the United States Department of State. In addition to the digitized text, we also make available and describe document-term matrices, which are datasets that systematically organize the word counts from each unique document by each unique term within the corpus of human rights documents. To contextualize the importance of this corpus, we describe the development of coding procedures in the human rights community and several existing categorical indicators that have been created by human coding of the human rights documents contained in the corpus. We then discuss how the new human rights corpus and the existing human rights datasets can be used with a variety of statistical analyses and machine learning algorithms to help scholars understand how human rights practices and reporting have evolved over time. We close with a discussion of our plans for dataset maintenance, updating, and availability.

Text mining techniques for the translation of personality questionnaires in cross-cultural research

Introduction There are guidelines and recommendations for translating and adapting tests to different languages to establish their validity and reliability. This work applied text mining techniques during the back-translation process of the ZKA-PQ across eleven cultures. Method Back-translated ZKA-PQ versions from eleven countries were analyzed. The back-translation process proceeded from the English language ZKA-PQ as the source, and the language/culture to which it was translated as the target questionnaire. Results There were comparisons among the items of the source and target versions, stopword removal, term-document matrices, synonym search, speech tagging, inter-term correlations, and a hierarchical clustering of the analyzed ZKA-PQ versions. Discussion Text mining can be of help in the analyses of the different versions of a self-report instrument when performing back-translation processes in cross-cultural research.

Maxent: An R Package for Low-memory Multinomial Logistic Regression with Support for Semi-automated Text Classification

maxent is a package with tools for data classification using multinomial logistic regression, also known as maximum entropy. The focus of this maximum entropy classifier is to minimize memory consumption on very large datasets, particularly sparse document-term matrices represented by the tm text mining package.

Updating the partial singular value decomposition in latent semantic indexing

Latent semantic indexing (LSI) is a method of information retrieval (IR) that relies heavily on the partial singular value decomposition (PSVD) of the term-document matrix representation of a data set. Calculating the PSVD of large term-document matrices is computationally expensive; hence in the case where terms or documents are merely added to an existing data set, it is extremely beneficial to update the previously calculated PSVD to reflect the changes. It is shown how updating can be used in LSI to significantly reduce the computational cost of finding the PSVD without significantly impacting performance. Moreover, it is shown how the computational cost can be reduced further, again without impacting performance, through a combination of updating and folding-in.

Compression, clustering, and pattern discovery in very high-dimensional discrete-attribute data sets

This paper presents an efficient framework for error-bounded compression of high-dimensional discrete-attribute data sets. Such data sets, which frequently arise in a wide variety of applications, pose some of the most significant challenges in data analysis. Subsampling and compression are two key technologies for analyzing these data sets. The proposed framework, PROXIMUS, provides a technique for reducing large data sets into a much smaller set of representative patterns, on which traditional (expensive) analysis algorithms can be applied with minimal loss of accuracy. We show desirable properties of PROXIMUS in terms of runtime, scalability to large data sets, and performance in terms of capability to represent data in a compact form and discovery and interpretation of interesting patterns. We also demonstrate sample applications of PROXIMUS in association rule mining and semantic classification of term-document matrices. Our experimental results on real data sets show that use of the compressed data for association rule mining provides excellent precision and recall values (above 90 percent) across a range of problem parameters while reducing the time required for analysis drastically. We also show excellent interpretability of the patterns discovered by PROXIMUS in the context of clustering and classification of terms and documents. In doing so, we establish PROXIMUS as a tool for both preprocessing data before applying computationally expensive algorithms and directly extracting correlated patterns.

Matrices with Low-Rank-Plus-Shift Structure: Partial SVD and Latent Semantic Indexing

We present a detailed analysis of matrices satisfying the so-called low-rank-plus-shift property in connection with the computation of their partial singular value decomposition (SVD). The application we have in mind is latent semantic indexing for information retrieval, where the term-document matrices generated from a text corpus approximately satisfy this property. The analysis is motivated by developing more efficient methods for computing and updating partial SVD of large term-document matrices and gaining deeper understanding of the behavior of the methods in the presence of noise.

Generating hierarchical document indices from common denominators in large document collections

This paper describes an effective, simple and efficient algorithm for computer generation of hierarchical indices from Document-Term matrices by means of calculating common denominator vectors from the document vector set. This procedure produces an intuitive, user-friendly hierarchical index of a document collection not unlike that which would be expected had a manual indexer set about to create an index or outline of a collection. The resulting index, when presented with a graphical user interface, provides the user with a natural easily comprehended view of the document collection that permits general browsing and informal search activities with an access method that requires no keyboard entry or prior knowledge of the vocabulary.

COMPUTING EXTREMAL SINGULAR TRIPLETS OF SPARSE MATRICES ON A SHARED-MEMORY MULTIPROCESSOR

In this paper, we develop four numerical methods for computing the singular value decomposition (SVD) of large sparse matrices on a shared-memory multiprocessor architecture. We particularly consider the SVD of unstructured sparse matrices in which the number of rows may be substantially larger or smaller than the number of columns. We emphasize Lanczos, block-Lanczos, subspace iteration and the trace minimization methods for determining a select number of smallest singular triplets (singular values and corresponding left- and right-singular vectors) for sparse matrices. The target architectures for implementations of such methods include the Alliant FX/80 and the Cray-2S/4–128. This algorithmic research is particularly motivated by recent information-retrieval techniques in which approximations to large sparse term-document matrices are needed, and by nonlinear inverse problems arising from seismic reflection tomography applications.

Large-Scale Sparse Singular Value Computations

We present four numerical methods for computing the singular value decomposition (SVD) of large sparse matrices on a multiprocessor architecture. We emphasize Lanczos and subspace iteration-based methods for determining several of the largest singular triplets (singular values and corresponding left- and right-singular vectors) for sparse matrices arising from two practical applications: information retrieval and seismic reflection tomography. The target architectures for our implementations are the CRAY-2S/4–128 and Alliant FX/80. The sparse SVD problem is well motivated by recent information-retrieval techniques in which dominant singular values and their corresponding singular vectors of large sparse term-document matrices are desired, and by nonlinear inverse problems from seismic tomography applications which require approximate pseudo-inverses of large sparse Jacobian matrices. This research may help advance the development of future out-of-core sparse SVD methods, which can be used, for example, to handle extremely large sparse matrices 0 × (106) rows or columns associated with extremely large databases in query-based information-retrieval applications.

Transitive reduction of a rectangular boolean matrix

We consider a problem of reducing rectangular boolean matrices. Some operations of reduction are defined and some interesting properties are obtained. The results are useful for models of information retrieval systems. Especially they are useful for simplification of document-term matrices.