Probabilistic Indexing Research Articles

Assessment of future drought sensitivity shift associated with crop yield loss risk in the Anthropocene

ABSTRACT This study proposed a framework using artificial neural networks (ANNs), the designed residual regression method, and probabilistic indexing, for drought sensitivity shift analysis, associated with future yield loss risk in Anthropocene. The temporal assessment of the drought index (standardized precipitation evapotranspiration index, SPEI) illustrated prevalent occurrence of mild to moderate drought with a fluctuating trend. Furthermore, the optimal ANN model, with R2 = 0.86 for training and 0.7 for testing, was used to project yield for future analysis. The projected spatial changes in drought sensitivity patterns suggest most of the area will experience an increased sensitivity to drought in the far future (2036–2050). The estimated probability loss under the different risk levels defined by the yield reduction rate was 6.2–55.5%. Furthermore, the spatial distribution of the designed risk index (IR) revealed the districts with the highest probability of vulnerability and showed a probable increase in future vulnerability.

Ground-truth generation through crowdsourcing with probabilistic indexes

Automatic transcription of large series of historical handwritten documents generally aims at allowing to search for textual information in these documents. However, automatic transcripts often lack the level of accuracy needed for reliable text indexing and search purposes. Probabilistic Indexing (PrIx) offers a unique alternative to raw transcripts. Since it needs training data to achieve good search performance, PrIx-based crowdsourcing techniques are introduced in this paper to gather the required data. In the proposed approach, PrIx confidence measures are used to drive a correction process in which users can amend errors and possibly add missing text. In a further step, corrected data are used to retrain the PrIx models. Results on five large series are reported which show consistent improvements after retraining. However, it can be argued whether the overall costs of the crowdsourcing operation pay off for the improvements, or perhaps it would have been more cost-effective to just start with a larger and cleaner amount of professionally produced training transcripts.

Open set classification of untranscribed handwritten text image documents

Content-based classification of manuscripts is an important task that is generally carried out by expert archivists. Nevertheless, many historical manuscript collections are so vast that in most cases this task is hardly feasible, even for large, well staffed archives. Nowadays, manuscripts are generally preserved in the form of sets of digital images. Therefore, the technical problem we are interested in is automatic classification of “‘image documents”, each consisting of a set of untranscribed handwritten text images, by the textual contents of the images. The traditional Pattern Recognition classification paradigm does provide the basic tools to deal with this problem. However, in practice, the set of relevant classes of a large documental series is seldom known in advance. Therefore, a classifier trained with a predefined set of classes will systematically fail when new image documents arrive which do not belong to any of the classes assumed in training. Here we adopt the “Open Set Classification” framework to extend and consolidate our previous work on image document classification in order to adequately handle new documents from unknown classes. The proposed approaches are based on a relatively novel technology for text image representation known as “probabilistic indexing”, which proves very effective to characterise the intrinsic word-level uncertainty exhibited by historical handwritten text images. We assess the performance of this approach on a moderately sized but representative dataset extracted from a huge series of complex notarial manuscripts from the Spanish Archivo Histórico Provincial de Cádiz, with good results.

Lexicon-based probabilistic indexing of handwritten text images

Keyword Spotting (KWS) is here considered as a basic technology for Probabilistic Indexing (PrIx) of large collections of handwritten text images to allow fast textual access to the contents of these collections. Under this perspective, a probabilistic framework for lexicon-based KWS in text images is presented. The presentation aims at providing formal insights which help understanding classical statements of KWS (from which PrIx borrows fundamental concepts), as well as the relative challenges entailed by these statements. The development of the proposed framework makes it clear that word recognition or classification implicitly or explicitly underlies any formulation of KWS. Moreover, it suggests that the same statistical models and training methods successfully used for handwriting text recognition can advantageously be used also for PrIx, even though PrIx does not generally require or rely on any kind of previously produced image transcripts. Experiments carried out using these approaches support the consistency and the general interest of the proposed framework. Results on three datasets traditionally used for KWS benchmarking are significantly better than those previously published for these datasets. In addition, good results are also reported on two new, larger handwritten text image datasets (Bentham and Plantas), showing the great potential of the methods proposed in this paper for indexing and textual search in large collections of untranscribed handwritten documents. Specifically, we achieved the following Average Precision values: IAMDB: 0.89, George Washington: 0.91, Parzival: 0.95, Bentham: 0.91 and Plantas: 0.92.

Processing a large collection of historical tabular images

Processing automatically historical document images to allow the search of textual information requires the preparation of ground-truth data for training and evaluation. This process is an expensive and arduous task, especially when the historical document images contain specialized vocabulary and/or tabular information. In the latter case, relevant decisions have to be taken to annotate the tabular parts. This paper presents a complex collection of historical document images and the resulting database, which is called HisClima. In this database, half of the images are in tabular format and half as running text. Both types of images contain pre-printed and handwritten text. The textual information is plenty of abbreviations and specific vocabulary related to weather conditions and old ships. This database can be used to research technologies related to historical document image processing and analysis, both for tabular and running text recognition. Baseline results are presented for Document Layout Analysis, Text Recognition, and Probabilistic Indexing. Although these results are good, there is still room for improvement and some indications are provided in this direction.

Sparse Topic Modeling: Computational Efficiency, Near-Optimal Algorithms, and Statistical Inference

Sparse topic modeling under the probabilistic latent semantic indexing (pLSI) model is studied. Novel and computationally fast algorithms for estimation and inference of both the word-topic matrix and the topic-document matrix are proposed and their theoretical properties are investigated. Both minimax upper and lower bounds are established and the results show that the proposed algorithms are rate-optimal, up to a logarithmic factor. Moreover, a refitting algorithm is proposed to establish asymptotic normality and construct valid confidence intervals for the individual entries of the word-topic and topic-document matrices. Simulation studies are carried out to investigate the numerical performance of the proposed algorithms. The results show that the proposed algorithms perform well numerically and are more accurate in a range of simulation settings comparing to the existing literature. In addition, the methods are illustrated through an analysis of the COVID-19 Open Research Dataset (CORD-19).

Extraction of The Spatio-temporal Activity Patterns Using Laser-scanner Trajectory Data

Abstract. A pedestrian tracking system on highly accurate laser scanners is an effective method to understand the usage of the facility space. While this system is capable of gathering an enormous volume of tracking data, specialized skills and significant amounts of labor are needed to get a reliable bird’s-eye view of the spatio-temporal characteristics of the observed data. In this paper, two methods to extract patterns of spatio-temporal activity are described. These can provide a broad overview of the office-worker’s activities in the office throughout a workday and an easily under-stood visualization that indicates what time segment, what location and what activities are taking place. One is a time segment extraction model that identifies characteristic time intervals in the time series data of office-worker’s activities using a classification model based on information loss minimization model. The other is a day scene extraction model that identifies daily scenes from simultaneous behavior patterns in spatio-temporal distributions using a latent class model with PLSI (Probabilistic latent semantic indexing). These methods provide viewpoints for separating their activities of a workday into time segments of appropriate size in order to obtain a grasp of how the activities vary with the time of day. Simultaneous behavior patterns in time, space, and activity are extracted, thereby allowing representation of typical scenes such as morning meetings and extended conversations between co-workers.

Hybrid Topic Cluster Models for Social Healthcare Data

Social media and in particular, microblogs are becoming an important data source for disease surveillance, behavioral medicine, and public healthcare. Topic Models are widely used in microblog analytics for analyzing and integrating the textual data within a corpus. This paper uses health tweets as microblogs and attempts the health data clustering by topic models. The traditional topic models, such as Latent Semantic Indexing (LSI), Probabilistic Latent Schematic Indexing (PLSI), Latent Dirichlet Allocation (LDA), Non-negative Matrix Factorization (NMF), and integer Joint NMF(intJNMF) methods are used for health data clustering; however, they are intractable to assess the number of health topic clusters. Proper visualizations are essential to extract the information from and identifying trends of data, as they may include thousands of documents and millions of words. For visualization of topic clouds and health tendency in the document collection, we present hybrid topic models by integrating traditional topic models with VAT. Proposed hybrid topic models viz., Visual Non-negative Matrix Factorization (VNMF), Visual Latent Dirichlet Allocation (VLDA), Visual Probabilistic Latent Schematic Indexing (VPLSI) and Visual Latent Schematic Indexing (VLSI) are promising methods for accessing the health tendency and visualization of topic clusters from benchmarked and Twitter datasets. Evaluation and comparison of hybrid topic models are presented in the experimental section for demonstrating the efficiency with different distance measures, include, Euclidean distance, cosine distance, and multi-viewpoint cosine similarity.

Probabilistic multi-word spotting in handwritten text images

Keyword spotting techniques are becoming cost-effective solutions for information retrieval in handwritten documents. We explore the extension of the single-word, line-level probabilistic indexing approach described in our previous works to allow for page-level search of queries consisting in Boolean combinations of several single-keywords. We propose heuristic rules to combine the single-word relevance probabilities into probabilistically consistent confidence scores of the multi-word boolean combinations. An empirical study, also presented in this paper, evaluates the search performance of word-pair queries involving AND and OR Boolean operations. Results of this study support the proposed approach and clearly show its effectiveness. Finally, a web-based demonstration system based on the proposed methods is presented.

A personalised user preference and feature based semantic information retrieval system in semantic web search

In this work, an ontology based Semantic Supported Information Retrieval System (SIRS) is introduced, in which the users give the input query which is determined by the Hypertext Markup Language (HTML) Parser then Probabilistic Latent Semantic Indexing (PLSI) algorithm is utilised to gathered the details in an effective manner. These conversations are preceded, along with the assistance of the field concept of the pre-existing domain ontologies, a mediator thesaurus and determine semantic association amongst them in nature. The proposed work concentrates on resolving the web search issues and also concentrates on resolving the personalised web search. So the SIRS will be correlated to the personalised search with item features, termed as Personalised User Preference and Feature based Semantic Information Retrieval System (PUFSIRS) architecture. PUFSIRS performs according to the particle agent, which accomplishes the SIRS based on the curiosity of the user through Multi-Criteria Particle Swarm Optimisation (MCPSO) for giving user's personal interest. The appropriate details for the semantic query are gathered and categorised based on the pertinence of an MCPSO procedure. The results show that the proposed PUFSIRS architecture can enhance the accuracy and efficiency for gathering the appropriate web-records which are collected from various fields such as food, education, news and healthcare to the current schemes. The result of the proposed PUFSIRS architecture is measured in terms of precision, recall, F-measure, accuracy and processing time.

An integrated robust semi-supervised framework for improving cluster reliability using ensemble method for heterogeneous datasets

Data mining literature offer some clustering techniques. But when we implement even an effective clustering technique, the results are found unreliable. The efficacy of the technique come under scrutiny. Here, the proposal is about an integrated framework, which ensures the reliability of the class labels assigned to a dataset whose class labels are unknown. The model uses PSO-k-means, k-medoids, c-means and Expectation Maximization for data clustering. This model integrates their results through majority voting cluster ensemble technique to enhance reliability. The reliable outcomes serve as the training set for the classification process through Bayesian classifier, Multi Layer Perceptron, Support Vector Machine and Decision tree. The predicted class labels by majority of classifiers through bagging classifier ensemble method are included with the training set and in combination, designated as the set with known class labels. Heterogeneous datasets with unknown class labels but known number of classes, after being treated through this model would be able to find the class labels for a significant portion of the data and may be accepted with reliability. The evaluation procedure has been performed by following the Dunn's, Davies–Bouldin and Modified Goodman–Kruskal indexing techniques for internal validation and probabilistic measures such as Normalized Mutual Information, Normalized Variation of Information and Adjusted Random Index which are appropriate measures of goodness-of-fit and robustness of the final clusters. The predictive capacity of the model is also validated through probabilistic measures and external indexing techniques such as Purity Measure, Random Index and F-measure.

Probabilistic latent semantic analysis applied to whole bacterial genomes identifies common genomic features.

The spread of drug resistance amongst clinically-important bacteria is a serious, and growing, problem [1]. However, the analysis of entire genomes requires considerable computational effort, usually including the assembly of the genome and subsequent identification of genes known to be important in pathology. An alternative approach is to use computational algorithms to identify genomic differences between pathogenic and non-pathogenic bacteria, even without knowing the biological meaning of those differences. To overcome this problem, a range of techniques for dimensionality reduction have been developed. One such approach is known as latent-variable models [2]. In latent-variable models dimensionality reduction is achieved by representing a high-dimensional data by a few hidden or latent variables, which are not directly observed but inferred from the observed variables present in the model. Probabilistic Latent Semantic Indexing (PLSA) is an extention of LSA [3]. PLSA is based on a mixture decomposition derived from a latent class model. The main objective of the algorithm, as in LSA, is to represent high-dimensional co-occurrence information in a lower-dimensional way in order to discover the hidden semantic structure of the data using a probabilistic framework. In this work we applied the PLSA approach to analyse the common genomic features in methicillin resistant Staphylococcus aureus, using tokens derived from amino acid sequences rather than DNA. We characterised genome-scale amino acid sequences in terms of their components, and then investigated the relationships between genomes and tokens and the phenotypes they generated. As a control we used the non-pathogenic model Gram-positive bacterium Bacillus subtilis.

An NMF-framework for Unifying Posterior Probabilistic Clustering and Probabilistic Latent Semantic Indexing

In document clustering, a document may be assigned to multiple clusters and the probabilities of a document belonging to different clusters are directly normalized. We propose a new Posterior Probabilistic Clustering (PPC) model that has this normalization property. The clustering model is based on Nonnegative Matrix Factorization (NMF) and flexible such that if we use class conditional probability normalization, the model reduces to Probabilistic Latent Semantic Indexing (PLSI). Systematic comparison and evaluation indicates that PPC is competitive with other state-of-art clustering methods. Furthermore, the results of PPC are more sparse and orthogonal, both of which are highly desirable.

Hierarchical Latent Semantic Class Extraction Using Asymmetric Term Similarities

Automatic extraction of term semantic classes is an interesting task in the Information Retrieval field. Models such as Latent Dirichlet Allocation or Probabilistic Semantic Indexing are able to provide the probability that a term belongs to a given semantic class. However such models do not provide neither Euclidean coordinates for terms nor a hierarchical structure to organize the latent semantic classes, which makes difficult to visualize the information under consideration. In this work we propose a hierarchical latent topic extraction method that exploits the information contained in asymmetric term similarity matrices. Our method produces Mercer Gram matrices for terms organized by frequency levels and then hierarchically combines classes belonging to different levels. Euclidean coordinates for terms can be recovered from the proposed kernel matrices. Our proposal also provides explicit conditional probabilities, as the Latent Dirichlet Allocation model does, but avoiding the computational burden usually involved in the iterative step of such probabilistic models. Finally, we analyze a real data base showing the advantages of the new approach.

Web Clustering Using Social Bookmark Data Regarding User Network

Social Bookmark services are web services which enable users to share their personal bookmark data with other users. In our research, we propose a web clustering method in a social bookmark service using bookmark data modified with user network data. Here, we define bookmark data as a pair of a user and a webpage he bookmarks, and user network data as links between users and their favorite users. In more detail, we use Probabilistic Latent Semantic Indexing (PLSI) which enables to project social bookmark data on latent semantic space and classify them by their semantic attributes. Using PLSI, we can classify webpages according to their topics, and regarding user network, we can append all bookmark data of user, who are reachable for a user, to his bookmark data. To confirm an effectiveness of our proposed clustering method, we have evaluated a clustering examination using real Social Bookmark Data. And finally we have confirmed that our proposed method could have classified webpages in the same topic.

Term suggestion with similarity measure based on semantic analysis techniques in query logs

PurposeTerm suggestion is a very useful information retrieval technique that tries to suggest relevant terms for users' queries, to help advertisers find more appropriate terms relevant to their target market. This paper aims to focus on the problem of using several semantic analysis methods to implement a term suggestion system.Design/methodology/approachThree semantic analysis techniques are adopted – latent semantic indexing (LSI), probabilistic latent semantic indexing (PLSI), and a keyword relationship graph (KRG) – to implement a term suggestion system.FindingsThis paper shows that using multiple semantic analysis techniques can give significant performance improvements.Research limitations/implicationsThe suggested terms returned from the system may be out of date, since the system uses a batch processing mode to update the training parameter.Originality/valueThe paper shows that the benefit of the techniques is to overcome the problems of synonymy and polysemy over the information retrieval field, by using a vector space model. Moreover, an intelligent stopping strategy is proposed to save the required number of iterations for probabilistic latent semantic indexing.

Looking back: On relevance, probabilistic indexing and information retrieval

Forty-eight years ago Maron and Kuhns published their paper, “On Relevance, Probabilistic Indexing and Information Retrieval” (1960). This was the first paper to present a probabilistic approach to information retrieval, and perhaps the first paper on ranked retrieval. Although it is one of the most widely cited papers in the field of information retrieval, many researchers today may not be familiar with its influence. This paper describes the Maron and Kuhns article and the influence that it has had on the field of information retrieval.

An Historical Note on the Origins of Probabilistic Indexing

The motivation behind “Probabilistic Indexing” was to replace two-valued thinking about information retrieval with probabilistic notions. This involved a new view of the information retrieval problem – viewing it as problem of inference and prediction, and introducing probabilistically weighted indexes and probabilistically ranked output. These ideas were first formulated and written up in August 1958.

Employing multiple representations for Chinese information retrieval

For information retrieval in the Chinese language, three representation methods for texts are popular, namely: 1-gram or character, bigram, and short-word. Each has its advantages as well as drawbacks. Employing more than one method may combine advantages from them and enhance retrieval effectiveness. We investigated two ways of using them simultaneously: mixing representations in documents and queries, and combining retrieval lists obtained via different representations. The experiments were done with the 170 MB evaluated Chinese corpora and 54 long and short queries available from the (TREC) program and using our Probabilistic Indexing and Retrieval Components System (PIRCS retrieval system). Experiments show that good retrieval need not depend on accurate word segmentation; approximate segmentation into short-words will do. Results also show and confirm that bigram representation alone works well; mixing characters with bigram representation boosts effectiveness further, but it is preferable to mix characters with short-word indexing which is more efficient, needs less resource, and gives better retrieval more often. Combining retrieval lists from short-word with character representation and from bigram indexing provides the best retrieval results but also at a substantial cost.

Probabilistic indexing for object recognition

Recent papers have indicated that indexing is a promising approach to fast model-based object recognition because it allows most of the possible matches between sets of image features and sets of model features to be quickly eliminated from consideration. This correspondence describes a system that is capable of indexing using sets of three points undergoing 3D transformations and projection by taking advantage of the probabilistic peaking effect. To be able to index using sets of three points, we must allow false negatives. These are overcome by ensuring that we examine several correct hypotheses. The use of these techniques to speed up the alignment method is described. Results are given on real and synthetic data.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>