Latent Semantic Indexing Method Research Articles

Markov Chains for User Queries Mining: Image Retrieval Application using Internet

Abstract: We suggest a brand-new technique for automatically annotating, indexing, and retrieving photos based on annotations. We introduce the novel technique, which we term Markovian Semantic Indexing (MSI), within the framework of an image retrieval system in online mode. In the event that such a system exists, the queries entered by the users are utilised to build an Aggregate Markov Chain (AMC), which defines the significance between the terms the system sees. The photographs are automatically annotated based on the queries entered by the users. Next, depending on the annotation of each image and the keyword relevance recorded in the AMC, a stochastic distance between them is presented. The suggested distance is given geometric meanings, and its relationship to a grouping in the keyword space is examined. The optimality qualities of the suggested distance are demonstrated by the use of a novel Markovian state similarity measure, the mean first cross passage time (CPT). Images are represented as points in a vector space, and MSI is used to calculate how similar two images are. In Annotation-Based Image Retrieval (ABIR) tasks, it is demonstrated that the novel method outperforms Latent Semantic Indexing (LSI) and probabilistic Latent Semantic Indexing (pLSI) methods in terms of Precision against Recall and has some theoretical advantages.

A scientometric analysis of birth cohorts in South Asia: Way forward for Pakistan.

The present study aims to: a) systematically map the of birth cohort studies from the South Asian region b) examine the major research foci and landmark contributions from these cohorts using reproducible scientometric techniques and c) offer recommendations on establishing new birth cohorts in Pakistan, building upon the strengths, weaknesses and gaps of previous cohorts. Bibliographic records for a total of 260 articles, published during through December 2018, were retrieved from the Web of Science (core database). All data were analysed using Microsoft Excel (2013), Web of Science platform and CiteSpace. A series of network analysis were then run for each time-period using the link reduction method and pathfinder network scaling. The co-cited articles were clustered into their homogeneous research clusters. The clusters were named using the Latent Semantic Indexing (LSI) method that utilized author keywords as source of names for these clusters. The scientometric analyses of original research output from these birth cohorts also paint a pessimistic landscape in Pakistan- where Pakistani sites for birth cohorts contributed only 31 publications; a majority of these utilized the MAL-ED birth cohort data. A majority of original studies were published from birth cohorts in India (156), Bangladesh (63), and Nepal (15). Out of these contributions, 31 studies reported data from multiple countries. The three major birth cohorts include prospective and multi-country MAL-ED birth cohort and The Pakistan Early Childhood Development Scale Up Trial, and a retrospective Maternal and infant nutrition intervention cohort. In addition to these, a few small-scale birth cohorts reported findings pertaining to neonatal sepsis, intrauterine growth retardation and its effects on linear growth of children and environmental enteropathy.

An efficient scheme for secure feature location using data fusion and data mining in internet of things environment

SummaryFeature location (FL) is performed to find the relationships between domain concepts and other software artifacts. One major problem in maintaining a software system is to understand how many functional features exist in a system and how these features are implemented. Also, poor security is the prime problem in the FL system. However, the existing recent FL techniques use a textual and dynamic approach, which is not found to be secure, keeping in view the changes in the description of security attacks. To overcome this drawback, this work proposed a novel secure approach for FL utilizing data fusion as well as data mining for the internet of things environment. Firstly, the repeated test cases (TC) are eradicated as of the labeled TC. Next, important attributes are selected using the artificial flora optimization algorithm from the removed labeled TC. Then, association rule mining is performed to ascertain closed attributes. Subsequently, encrypt the closed attributes utilizing Caesar Cipher‐Rivest, Shamirs, as well as Adelman algorithm. After that, the score value of the closed attributes counts was found utilizing entropy calculation. Finally, the score value is given as input to the normalized‐K‐Means (N‐[K‐Means]) algorithm, where the score value is normalized utilizing min‐max normalization and then grouped utilizing K‐Means algorithm (KMA). It proffers better results for FL in the source code. The proposed N‐(K‐Means) performance is found better in comparison to the KMA and latent semantic indexing methods. The proposed system proffered better FL results in comparison to the other prevailing methods.

Artificial intelligence snapchat: Visual conversation agent

Visual conversation is a dialog in which parties exchange visual information. The key novelty presented in this paper is an artificial intelligence-driven visual conversation automation method. We will present a state of the art Artificial Intelligence Snapchat Visual Conversation Agent (AISVCA). AISVCA uses our proposed artificial intelligence-driven visual conversation automation method to create received image caption and generate an appropriate reasonable visual response. These functionalities are achieved by using a combination of Convolutional Neural Network (CNN), Long Short-Term Memory Neural Network (LSTM) and, Latent Semantic Indexing method (LSI). CNN and LSTM are used to create image captions and, LSI is used to assess the semantic similarity between captions generated from personalized image dataset, and captions that are extracted from the received image content. We will show that AISVCA, using the proposed method can generate a visual response that is basically indistinguishable from a human visual response. To evaluate the proposed approach, we measured the accuracy of the proposed system and, conducted a user study to test communication quality. In the user study, we analyzed source credibility and interpersonal attraction of the AISVCA. The user study results showed that there are no significant differences in communication quality between a visual conversation with AISVCA and visual conversation with the human agent.

A Hybrid Model for Topic Modeling Using Latent Dirichlet Allocation and Feature Selection Method

In this information age, Knowledge discovery and pattern matching plays a significant role. Topic Modeling, an area of Text mining is used detecting hidden patterns in a document collection. Topic Modeling and Document Clustering are two important key terms which are similar in concepts and functionality. In this paper, topic modeling is carried out using Latent Dirichlet Allocation-Brute Force Method (LDA-BF), Latent Dirichlet Allocation-Back Tracking (LDA-BT), Latent Semantic Indexing (LSI) method and Nonnegative Matrix Factorization (NMF) method. A hybrid model is proposed which uses Latent Dirichlet Allocation (LDA) for extracting feature terms and Feature Selection (FS) method for feature reduction. The efficiency of document clustering depends upon the selection of good features. Topic modeling is performed by enriching the good features obtained through feature selection method. The proposed hybrid model produces improved accuracy than K-Means clustering method.

Latent semantic indexing (LSI) and its impact on copywriting

Latent semantic indexing (LSI) is becoming more and more popular in copywriting, gradually replacing texts written on the principles of SEO. LSI was called in the 2010s, when popular search engines switched to a qualitatively new way of ranking materials and sites. The difference between SEO and LSI ways of creation lies in the fact that search engines rank SEO materials by keywords, while LSI are ranked how fully the topic is covered and how useful the article will be to the reader. Consequently, in addition to keywords and phrases, the associative core is involved here. Materials written for people have replaced the texts created for the search engine. The article describes the algorithm for creation of the associative and thematic core, the ways in which this can be done. The basic steps helping to create an LSI text are also shown.The author underlines that due to the specificity of the presentation of a significant amount of information and the maximum expertise in the disclosure of the topic, text writers accustomed to working on the principles of SEO have to learn to write within a new paradigm. The owners of the websites that host articles created by LSI principles have discovered the advantages of this way of presenting information, since their resources have become better indexed and take the leading positions in search results. Such algorithms as “Baden-Baden”, “Korolev” and “Panda” have positively influenced the Internet environment as a whole, since re-optimized texts, which were filled with keys and were of little use to the reader, now have turned out to be on the last positions of issue. The new method of ranking according to the LSI method allows specialists to create the texts that are not only useful and expert but also differ in lexical richness, using expressive and figurative means of the language, which could not be assumed in SEO materials.It is highlighted in the article the use of neural networks should bring the way of presenting information to the consumer’s needs even more, inventing techniques that will allow leading materials created in an ordinary language to lead the positions without the need to incorporate key phrases into the text. We believe that the LSI-method, which has perfectly manifested itself in copywriting, is capable of unlocking the potential of the media texts, which are now being written on the principles of SEO.

Model detecting learning styles with artificial neural network

Currently the detection of learning styles from the external aspect has not produced optimal results. This research tries to solve the problem by using an internal approach. The internal approach is one that derives from the personality of the learner. One of the personality traits that each learner possesses is prior knowledge. This research starts with the prior knowledge generation process using the Latent Semantic Indexing (LSI) method. LSI is a technique using Singular Value Decomposition (SVD) to find meaning in a sentence. LSI works to generate the prior knowledge of each learner. After the prior knowledge is raised, then one can predict learning style using the artificial neural network (ANN) method. The results of this study are more accurate than the results of detection conducted with an external approach.

Latent Semantic Indexing for Indonesian Text Similarity

Document is a written letter that can be used as evidence of information. Plagiarism is a deliberate or unintentional act of obtaining or attempting to obtain credit or value for a scientific work, citing some or all of the scientific work of another party acknowledged as a scientific work without stating the source properly and adequately. Latent Semantic Indexing method serves to find text that has the same text against from a document. The algorithm used is TF/IDF Algorithm that is the result of multiplication of TF value with IDF for a term in document while Vector Space Model (VSM) is method to see the level of closeness or similarity of word by way of weighting term.

Analysis on the use of Latent Semantic Indexing (LSI) for document classification and retrieval system of PNP files

Document classification is the process of categorizing documents from many mixed files automatically [1]. In this paper, an approach to classification of documents for admin-case files of Philippine National Police (PNP) using Latent Semantic Indexing (LSI) method is proposed. The model for this that represents term-to-term, document-todocument and term-to-document relationships has been applied. Regular Expression is implemented also to define a search pattern based on character strings which the LSI used to establish the semantic relevance of the character strings to the search term or keyword. The aim of the study is to evaluate the performance of LSI in classifying PNP documents; experimentation was done using software to test the capability of LSI towards text retrieval. Indexing is according to the pattern matched in the collection of text that uses model of SVD. Based on tests, documents were indexed based on file relationships and was able to return a search result as the retrieved information from PNP files. Weights are used to check the accuracy of the method; the positive values identified in query similarity are regarded as the most relevant among the related searches, meaning, the query word matches words in a text file and it returns a query result.

Beyond vector space model for hierarchical Arabic text classification: A Markov chain approach

The vector space model (VSM) is a textual representation method that is widely used in documents classification. However, it remains to be a space-challenging problem. One attempt to alleviate the space problem is by using dimensionality reduction techniques, however, such techniques have deficiencies such as losing some important information. In this paper, we propose a novel text classification method that neither uses VSM nor dimensionality reduction techniques. The proposed method is a space efficient method that utilizes the first order Markov model for hierarchical Arabic text classification. For each category and sub-category, a Markov chain model is prepared based on the neighboring characters sequences. The prepared models are then used for scoring documents for classification purposes. For evaluation, we used a hierarchical Arabic text data collection that contains 11,191 documents that belong to eight topics distributed into 3-levels. The experimental results show that the Markov chains based method significantly outperforms the baseline system that employs the latent semantic indexing (LSI) method. That is, the proposed method enhances the F1-measure by 3.47%. The novelty of this work lies on the idea of decomposing words into sequences of characters, which found to be a promising approach in terms of space and accuracy. Based on our best knowledge, this is the first attempt to conduct research for hierarchical Arabic text classification with such relatively large data collection.

Accelerating large partial EVD/SVD calculations by filtered block Davidson methods

Partial eigenvalue decomposition (PEVD) and partial singular value decomposition (PSVD) of large sparse matrices are of fundamental importance in a wide range of applications, including latent semantic indexing, spectral clustering, and kernel methods for machine learning. The more challenging problems are when a large number of eigenpairs or singular triplets need to be computed. We develop practical and efficient algorithms for these challenging problems. Our algorithms are based on a filter-accelerated block Davidson method. Two types of filters are utilized, one is Chebyshev polynomial filtering, the other is rational-function filtering by solving linear equations. The former utilizes the fastest growth of the Chebyshev polynomial among same degree polynomials; the latter employs the traditional idea of shift-invert, for which we address the important issue of automatic choice of shifts and propose a practical method for solving the shifted linear equations inside the block Davidson method. Our two filters can efficiently generate high-quality basis vectors to augment the projection subspace at each Davidson iteration step, which allows a restart scheme using an active projection subspace of small dimension. This makes our algorithms memory-economical, thus practical for large PEVD/PSVD calculations. We compare our algorithms with representative methods, including ARPACK, PROPACK, the randomized SVD method, and the limited memory SVD method. Extensive numerical tests on representative datasets demonstrate that, in general, our methods have similar or faster convergence speed in terms of CPU time, while requiring much lower memory comparing with other methods. The much lower memory requirement makes our methods more practical for large-scale PEVD/PSVD computations.

Augmenting matrix factorization technique with the combination of tags and genres

Abstract Recommender systems play an important role in our daily life and are becoming popular tools for users to find what they are really interested in. Matrix factorization methods, which are popular recommendation methods, have gained high attention these years. With the rapid growth of the Internet, lots of information has been created, like social network information, tags and so on. Along with these, a few matrix factorization approaches have been proposed which incorporate the personalized information of users or items. However, except for ratings, most of the matrix factorization models have utilized only one kind of information to understand users’ interests. Considering the sparsity of information, in this paper, we try to investigate the combination of different information, like tags and genres, to reveal users’ interests accurately. With regard to the generalization of genres, a constraint is added when genres are utilized to find users’ similar “soulmates”. In addition, item regularizer is also considered based on latent semantic indexing (LSI) method with the item tags. Our experiments are conducted on two real datasets: Movielens dataset and Douban dataset. The experimental results demonstrate that the combination of tags and genres is really helpful to reveal users’ interests.

Using SVD on Clusters to Improve Precision of Interdocument Similarity Measure.

Recently, LSI (Latent Semantic Indexing) based on SVD (Singular Value Decomposition) is proposed to overcome the problems of polysemy and homonym in traditional lexical matching. However, it is usually criticized as with low discriminative power for representing documents although it has been validated as with good representative quality. In this paper, SVD on clusters is proposed to improve the discriminative power of LSI. The contribution of this paper is three manifolds. Firstly, we make a survey of existing linear algebra methods for LSI, including both SVD based methods and non-SVD based methods. Secondly, we propose SVD on clusters for LSI and theoretically explain that dimension expansion of document vectors and dimension projection using SVD are the two manipulations involved in SVD on clusters. Moreover, we develop updating processes to fold in new documents and terms in a decomposed matrix by SVD on clusters. Thirdly, two corpora, a Chinese corpus and an English corpus, are used to evaluate the performances of the proposed methods. Experiments demonstrate that, to some extent, SVD on clusters can improve the precision of interdocument similarity measure in comparison with other SVD based LSI methods.

A two-stage feature selection method for text categorization by using category correlation degree and latent semantic indexing

With the purpose of improving the accuracy of text categorization and reducing the dimension of the feature space, this paper proposes a two-stage feature selection method based on a novel category correlation degree (CCD) method and latent semantic indexing (LSI). In the first stage, a novel CCD method is proposed to select the most effective features for text classification, which is more effective than the traditional feature selection method. In the second stage, document representation requires a high dimensionality of the feature space and does not take into account the semantic relation between features, which leads to a poor categorization accuracy. So LSI method is proposed to solve these problems by using statistically derived conceptual indices to replace the individual terms which can discover the important correlative relationship between features and reduce the feature space dimension. Firstly, each feature in our algorithm is ranked depending on their importance of classification using CCD method. Secondly, we construct a new semantic space based on LSI method among features. The experimental results have proved that our method can reduce effectively the dimension of text vector and improve the performance of text categorization.

Geospatial Information Retrieval Base on Query Expansion and Semantic Indexing

Accepted: 24.11.2015 In this paper, a hybrid approach is developed for geospatial information retrieval which combines query expansion (QE) and latent semantic indexing (LSI) methods. The hybrid method uses the advantages of Query Expansion and Semantic indexing methods for improving search results. LSI establishes relations depending on the similarities between queries and the documents where QE helps by adding extra similar terms to the queries. The dataset is populated using data extracted from Wikipedia and USGS (The United States Geological Survey). Automation is programmed to get all the information from the web and to extract the meaningful words. The significant terms are the name of the places, directions, and nearby locations. The dataset includes mountains and places with their latitudes, longitudes, locations and directions. The extracted data is also taken into Protege for semantically querying. The approach is applied to geographical data (popular search items on search engines), and by querying the data with the combination of LSI and QE more related results are handled than the results of methods when they are applied alone. Many queries are run, and the results are listed and compared. At the end of the study it is understood that the hybrid method considerably improves the search results.

Text Categorization Comparison between Simple BPNN and Combinatorial Method of LSI and BPNN

paper proposed a text categorization comparison between simple BPNN and Combinatorial method of LSI and BPNN. In the traditional error back propagation network, the weight adjustment process gets block in a local minima and also the training speed of such network is very slow which leads to reduced performance and reduced efficiency of the network. Also the Learning time of overall network is very high.Hence, to improve the categorization accuracy, a new combinatorial method of LSI (latent semantic Indexing) and BPNN (back propagation neural network) is proposed. The latent semantics demonstration is an accurate data structure in low-dimensional space in which documents, terms and queries are rooted and also compared. Singular value decomposition (SVD) technique is used in Latent semantic Analysis in which large term-document matrix is decomposed into a set of k orthogonal factors by which the original textual data is changed to a smaller semantic space. New document vectors are found in reduced k-dimensional space. Also new coordinates of the queries are found. Here we implement combinatorial method of LSI and BPNN based technique for the classification of 20Newsgroup dataset which include categories of Sports, CS, and Medicine. The proposed technique implemented is compared with the existing BPNN technique. . Hence, this new method greatly reduces the dimension and better classification results can be achieved.

Text classification using genetic algorithm oriented latent semantic features

In this paper, genetic algorithm oriented latent semantic features (GALSF) are proposed to obtain better representation of documents in text classification. The proposed approach consists of feature selection and feature transformation stages. The first stage is carried out using the state-of-the-art filter-based methods. The second stage employs latent semantic indexing (LSI) empowered by genetic algorithm such that a better projection is attained using appropriate singular vectors, which are not limited to the ones corresponding to the largest singular values, unlike standard LSI approach. In this way, the singular vectors with small singular values may also be used for projection whereas the vectors with large singular values may be eliminated as well to obtain better discrimination. Experimental results demonstrate that GALSF outperforms both LSI and filter-based feature selection methods on benchmark datasets for various feature dimensions.

Fast Updating Algorithms for Latent Semantic Indexing

This paper discusses a few algorithms for updating the approximate singular value decomposition (SVD) in the context of information retrieval by latent semantic indexing (LSI) methods. A unifying framework is considered which is based on Rayleigh--Ritz projection methods. First, a Rayleigh--Ritz approach for the SVD is discussed and it is then used to interpret the Zha and Simon algorithms [SIAM J. Sci. Comput., 21 (1999), pp. 782--791]. This viewpoint leads to a few alternatives whose goal is to reduce computational cost and storage requirement by projection techniques that utilize subspaces of much smaller dimension. Numerical experiments show that the proposed algorithms yield accuracies comparable to those obtained from standard ones at a much lower computational cost.

Mining User Queries with Markov Chains: Application to Online Image Retrieval

We propose a novel method for automatic annotation, indexing and annotation-based retrieval of images. The new method, that we call Markovian Semantic Indexing (MSI), is presented in the context of an online image retrieval system. Assuming such a system, the users' queries are used to construct an Aggregate Markov Chain (AMC) through which the relevance between the keywords seen by the system is defined. The users' queries are also used to automatically annotate the images. A stochastic distance between images, based on their annotation and the keyword relevance captured in the AMC, is then introduced. Geometric interpretations of the proposed distance are provided and its relation to a clustering in the keyword space is investigated. By means of a new measure of Markovian state similarity, the mean first cross passage time (CPT), optimality properties of the proposed distance are proved. Images are modeled as points in a vector space and their similarity is measured with MSI. The new method is shown to possess certain theoretical advantages and also to achieve better Precision versus Recall results when compared to Latent Semantic Indexing (LSI) and probabilistic Latent Semantic Indexing (pLSI) methods in Annotation-Based Image Retrieval (ABIR) tasks.

Latent Text Mining for Cybercrime Forensics

Recent research reveals that the number of cyber-attacks has been doubled in the past three years. This is a devastating growth of the number of cyber-attacks, and it reveals a serious business problem around the world. Existing intrusion detection systems (IDSs), intrusion prevention systems (IPSs), and anti-malware systems mainly rely on low-level network traffic features or program code signatures to detect cyber-attacks. However, since hackers can constantly change their attack tactics by, it is extremely difficult for existing security solutions to detect cyber-attacks. There are increasing more evidences showing that cybercriminals tend to exchange cybercrime knowledge and transact via online social media. Accordingly, it presents unprecedented opportunities for security intelligence experts to tap into online social media to extract the vital security intelligence for cyber-attack forensics. The main contributions of this paper are the design, development, and evaluation of a Latent Dirichlet Allocation (LDA)-based latent text mining model for cyber-attack forensics. Our preliminary evaluation of the proposed latent text mining model based on a real-world data set crawled from Twitter and Blog sites shows that it significantly outperforms the probabilistic latent semantic indexing (pLSI) method in terms of extracting more relevant and richer concepts describing real-world cyber-attack incidents.