Full-text Search Engine Research Articles

The next-gen federated search architecture for biomedical knowledge repositories — The LIT-FED-SEARCH engine

The primary objective of LIT-FED-SEARCH software is to develop a user-friendly solution tailored to researchers and scientists. This solution aims to enhance their impact by facilitating the analysis of data from modern, extensive datasets like PubMed and Clinical Trials, alongside real-world evidence. The central concept we offer is a Federated Search Workflow Engine, which has been designed and maintained to accommodate various infrastructure configurations for the convenience of users. In line with this approach, potential users have the flexibility to configure their own computing environment and a set of interesting data repositories, based on their specific requirements and capabilities. This customization can significantly reduce the time and resources invested in research. LIT-FED-SEARCH is constructed with the support of OpenSearch full-text search engine as its heart. This paper offers an overview of the system’s architecture, capabilities, and potential applications in the field of biomedical research.

Big Data Analysis using Elasticsearch and Kibana: A Rating Correlation to Sustainable Sales of Electronic Goods

Big data collection involves enormous amounts of raw data. To boost the sustainability of corporate value and support business intelligence and decision-making systems, in-depth data analysis is necessary. The data storage, analysis, and visualization methods, as well as the discovery of patterns and linkages, all depend on extensive data analysis. This study aims to process datasets to learn things like how ratings impact market sales transactions and how much of an impact factor connected to consumers and items have on ratings. Elasticsearch and Kibana were used for the dataset processing. This study evaluated traits related to the test parameters using a variety of test procedures. The product is scored as a representation of the product types involved in the sales transaction, and the name is assessed as a reflection of the customer. Kibana and Elasticsearch, a full-text search engine, were used in this work to do extensive data analysis on data sets. It is a visualization tool that is employed in a controlled environment to evaluate how ratings impact market exchanges for electronic goods, and it offers suggestions. The study found a substantial relationship between electronic product sales on the Amazon marketplace from 2012 to 2018. It suggested the importance of buyer constituents as users and how different product categories relate to ratings in business transactions. Doi: 10.28991/HIJ-2023-04-03-09 Full Text: PDF

A comparison of Apache Solr and Elasticsearch technologies in support of large-scale data analysis

In the era of big data, data has never been more important because it contains hidden insights. Additionally, it is necessary and challenging to extract usable information from enormous volumes of data. When attempting to perform data processing and analytics in a variety of domains, developers of data-intensive systems have consequently met several challenges. In addition, full-text search is one of the most significant components of big data processing and analytics for discovering fragments of required data among large volumes of data. Due to the importance of the subject, this article begins with an examination of the characteristics, capabilities, and technical comparisons of full-text search technologies, followed by a systematic comparison of Apache Solr and Elasticsearch in terms of indexing times and queries on three separate datasets. According to our findings, based on default configuration, Apache Solr has better performance when looking at indexing times measured on three machines with different hardware specifications. Likewise, Apache Solr outperforms Elasticsearch in seven out of ten search queries. Regarding our results, on computers with restricted hardware resources, we recommend utilizing Apache Solr instead of Elasticsearch. In addition, this study provides researchers and developers of data-intensive systems with a complete comparison and suggestions for choosing the most effective full-text search engine for their task.

Corporate Governance Data Storage Tort Liability for Artificial Intelligence and Emotion Recognition

The rapid development of information technology in the 21st century has realized the transformation of people’s production and lifestyle. At the same time, the current research on data infringement is increasing. At present, the legal field has gradually begun to apply informatization, using its advanced data exploration methods and collection technologies. This paper aimed to realize the research on tort liability of corporate governance data storage under the background of artificial intelligence and emotion recognition, so as to promote the ecological development of tort liability and the protection of rights and interests in the era of big data. The principles and laws of neural network technology in artificial intelligence were used for reference to promote the development of data infringement tracking technology. In the experiment of numerical information extraction under the neural network, the recognition rate of the system algorithm only exceeded 98% in time, and the highest recognition rate in location type was only 95.39%. Therefore, it showed that the method proposed in this paper achieved a certain recall rate, and the method was flexible and feasible. Due to the limited training sample size, the accuracy of subjects and attributes was relatively low. However, accuracy topics and attributes had relatively little impact on search results. The subsequent search module was based on a full-text search engine. Therefore, it is of great significance to use neural network technology to study the tort liability of data storage in the context of corporate governance.

Search and sequence analysis tools services from EMBL-EBI in 2022.

The EMBL-EBI search and sequence analysis tools frameworks provide integrated access to EMBL-EBI’s data resources and core bioinformatics analytical tools. EBI Search (https://www.ebi.ac.uk/ebisearch) provides a full-text search engine across nearly 5 billion entries, while the Job Dispatcher tools framework (https://www.ebi.ac.uk/services) enables the scientific community to perform a diverse range of sequence analysis using popular bioinformatics applications. Both allow users to interact through user-friendly web applications, as well as via RESTful and SOAP-based APIs. Here, we describe recent improvements to these services and updates made to accommodate the increasing data requirements during the COVID-19 pandemic.

Intelligent Resume Retrieval Based on Lucence

With the development of Internet, the electronic resume has gradually replaced the paper one. It is the basic requirement of recruitment for enterprises to retrieve the talent information that fulfills the requirement quickly and without omission.Based on the framework of SpringBoot and Lucence full-text search engine, this paper implements a resume intelligent filtering algorithm, which improves the query speed of the system by establishing an index database. At the same time,the scoring function improves the accuracy of the filtering results, reduces the pressure of high concurrency of the database, improves the work efficiency of the Human Resources Department, and avoids the talent loss.

Full-text search engine with suffix index for massive heterogeneous data

Existing popular search engines like Elasticsearch (ES) commonly use inverted indices to quickly retrieve source data matching a given set of queries. However, an inverted index may not find all of the matching results from data, particularly those that are hard to be segmented into words, such as data logs and scientific signals. This article presents our innovative technique for a true full-text search system called SAES by replacing the inverted index in ES with the suffix index to guarantee a 100% recall ratio. We designed a distributed suffix index scheme with online building and offline merging capable of scaling with the architecture of ES. The suffix index is dynamically constructed by several suffix array construction tools which adapt to the data size and available computing resources such as CPU cores, RAM, and disk capacities. Furthermore, it can be compacted to provide a trade-off between searching speed and index storage space. An experimental study was conducted to test the functions and performance of single- and multi-node SAES on realistic datasets of texts, logs, genomes, and signals. The systems performed well for both exact and approximate search queries defined on units of bytes or half-bytes. This work provides a feasible reference design for extending ES with suffix index to support true full-text searches over massive heterogeneous data.

The information retrieval anthology 2021

The Information Retrieval Anthology, IR Anthology for short, is an endeavor to create a comprehensive collection of metadata and full texts of IR-related publications. We report on its first release, the use cases it can serve, as well as the challenges lying ahead to develop it towards a resource that serves the IR community for years to come. The IR Anthology's metadata browser and full text search engine are available at IR.webis.de.

Design and analysis of intelligent retrieval system for drilling data and completion data based on cloud platform

In the context of the accelerating cloud platform construction in the oil industry, this paper solves the problem of difficulty in drilling and completion data retrieval by classifying and sorting drilling data and completion data to construct an intelligent retrieval model for drilling and completion data. Based on the cloud platform environment, this paper studies the method of constructing an retrieval system for drilling data and completion data through the microservice architecture. This system provides a unified retrieve page for structured data and unstructured data of drilling and completion operations. By using the ElasticSearch full-text search engine to build a retrieval model, it provides users with fast, efficient, and convenient comprehensive query and retrieval services, and lays a foundation for the intelligent construction of drilling and completion platforms.

Ring: Real-Time Emerging Anomaly Monitoring System Over Text Streams

Microblog platforms have been extremely popular in the big data era due to its real-time diffusion of information. It's important to know what anomalous events are trending on the social network and be able to monitor their evolution and find related anomalies. In this paper we demonstrate Ring , a r eal-t i me emerging a n omaly monitorin g system over microblog text streams. Ring integrates our efforts on both emerging anomaly monitoring research and system research. From the anomaly monitoring perspective, Ring proposes a graph analytic approach such that (1) Ring is able to detect emerging anomalies at an earlier stage compared to the existing methods, (2) Ring is among the first to discover emerging anomalies correlations in a streaming fashion, (3) Ring is able to monitor anomaly evolutions in real-time at different time scales from minutes to months. From the system research perspective, Ring (1) optimizes time-ranged keyword query performance of a full-text search engine to improve the efficiency of monitoring anomaly evolution, (2) improves the dynamic graph processing performance of Spark and implements our graph stream model on it, As a result, Ring is able to process big data to the entire Weibo or Twitter text stream with linear horizontal scalability. The system clearly presents its advantages over existing systems and methods from both the event monitoring perspective and the system perspective for the emerging event monitoring task.

Performance Analysis of Elastic Search Technique in Identification and Removalof Duplicate Data

Elastic search is a way to organize the data and make it easily accessible. It is a server based search on Lucene. It is a highly scalable, distributed and full-text search engine. Elastic search is developed in Java. It is published as open source under the terms of the Apache License. Elastic search is the most popular enterprise search engine. Elastic search includes all advances in speed, security, scalability, and hardware efficiency. Elastic search is a tool for querying written words. It can perform some other smart tasks, but its principal is returning text similar to a given query and statistical analyses of a quantity of text. Elasticsearch is a standalone database server, which is written in Java and using HTTP/JSON protocol,it’s takes data and optimized the data according to language based searches and stores it in a sophisticated format. Elastic search is very convenient, supporting clustering and leader selection out of the box. Whether it’s searching a database of trade products by description, finding similar text in a body of crawled web pages. In this manuscript elastic search capability of copied data identification and its removing techniques performance are analyzed.

Application of full-text search engine in coal mine operation and maintenance system

Application of full-text search engine in coal mine operation and maintenance system

An intelligent query approach by integrating visual queries with full-text search

This research focuses on a combined query approach for handling structured and unstructured data. A Visual Query System (VQS) is usually developed for handling structured data. It enables end users to formulate queries through visual representations without the need of having knowledge of database and query languages. As an important branch of today's information retrieval technology, Full-Text Search (FTS) is an important approach for dealing with unstructured data. Hence, a hybrid system can be developed, which integrates a Visual Query System with a Full-Text Search engine and which acts as an intelligent search interface. A customized ranking algorithm can rank the results according to their relevance. This hybrid approach converts the natural language text queries and visual queries into a database query language, such as SQL. It can be adopted for different examples of data with a combination of structured and unstructured types to provide an efficient query interface solution. In particular, it can be integrated in web search engines to integrate structured data with unstructured data in the search interface.

A high-performance and flexible chemical structure & data search engine built on CouchDB & ElasticSearch

Computer-assisted chemical structure searching plays a critical role for efficient structure screening in cheminformatics. We designed a high-performance chemical structure & data search engine called DCAIKU, built on CouchDB and ElasticSearch engines. DCAIKU converts the chemical structure similarity search problem into a general text search problem to utilize off-the-shelf full-text search engines. DCAIKU also supports flexible document structures and heterogeneous datasets with the help of schema-less document database. Our evaluations show that DCAIKU can handle both keyword search and structural search against millions of records with both high accuracy and low latency. We expect that DCAIKU will lay the foundation towards large-scale and cost-effective structural search in materials science and chemistry research.

20 years of the SMART protein domain annotation resource.

SMART (Simple Modular Architecture Research Tool) is a web resource (http://smart.embl.de) for the identification and annotation of protein domains and the analysis of protein domain architectures. SMART version 8 contains manually curated models for more than 1300 protein domains, with approximately 100 new models added since our last update article (1). The underlying protein databases were synchronized with UniProt (2), Ensembl (3) and STRING (4), doubling the total number of annotated domains and other protein features to more than 200 million. In its 20th year, the SMART analysis results pages have been streamlined again and its information sources have been updated. SMART’s vector based display engine has been extended to all protein schematics in SMART and rewritten to use the latest web technologies. The internal full text search engine has been redesigned and updated, resulting in greatly increased search speed.

Design and Implementation of Full Text Search Engine Based on Lucene

With the development of information technology, the types and quantities of documents are increasing rapidly. How can people extract information that they need from these documents quickly and accurately, which have already become a very important problem? For this problem, this paper proposed the full text search engine based on Lucene. At first, it achieved the extraction of text content based on the various types of documents, and established indexes for the various types of documents under the folder, then denoted the keyword by the logical expressions. Finally, we used the GUI (Graphical User Interface) to classify and display the result of retrieval, and verified the processes of establishing index and retrieving through experiments. Experimental results show that Lucene can retrieve the documents quickly and accurately, so meet the needs of users.

KrillDB: A de novo transcriptome database for the Antarctic krill (Euphausia superba).

Antarctic krill (Euphausia superba) is a key species in the Southern Ocean with an estimated biomass between 100 and 500 million tonnes. Changes in krill population viability would have catastrophic effect on the Antarctic ecosystem. One looming threat due to elevated levels of anthropogenic atmospheric carbon dioxide (CO2) is ocean acidification (lowering of sea water pH by CO2 dissolving into the oceans). The genetics of Antarctic krill has long been of scientific interest for both for the analysis of population structure and analysis of functional genetics. However, the genetic resources available for the species are relatively modest. We have developed the most advanced genetic database on Euphausia superba, KrillDB, which includes comprehensive data sets of former and present transcriptome projects. In particular, we have built a de novo transcriptome assembly using more than 360 million Illumina sequence reads generated from larval krill including individuals subjected to different CO2 levels. The database gives access to: 1) the full list of assembled genes and transcripts; 2) their level of similarity to transcripts and proteins from other species; 3) the predicted protein domains contained within each transcript; 4) their predicted GO terms; 5) the level of expression of each transcript in the different larval stages and CO2 treatments. All references to external entities (sequences, domains, GO terms) are equipped with a link to the appropriate source database. Moreover, the software implements a full-text search engine that makes it possible to submit free-form queries. KrillDB represents the first large-scale attempt at classifying and annotating the full krill transcriptome. For this reason, we believe it will constitute a cornerstone of future approaches devoted to physiological and molecular study of this key species in the Southern Ocean food web.

Merged Ontology and SVM-Based Information Extraction and Recommendation System for Social Robots

The recent technology of human voice capture and interpretation has spawned the social robot to convey information and to provide recommendations. This technology helps people obtain information about a particular topic after giving an oral query to a humanoid robot. However, most of the search engines are keyword-matching mechanism-based, and the existing full-text query search engines are inadequate at retrieving relevant information from various oral queries. With only predefined words and sentence-based recommendations, a social robot may not suggest the correct items, if items retrieved along with the information are not predefined. In addition, the available conventional ontology-based systems cannot extract precise data from webpages to show the correct results. In this regard, we propose a merged ontology and support vector machine (SVM)-based information extraction and recommendation system. In the proposed system, when a humanoid robot receives an oral query from a disabled user, the oral query changes into a full-text query, the system mines the full-text query to extract the disabled user’s needs, and then converts the query into the correct format for a search engine. The proposed system downloads a collection of information about items (city features, diabetes drugs, and hotel features). The SVM identifies the relevant information on the item and removes anything irrelevant. Merged ontology-based sentiment analysis is then employed to find the polarity of the item for recommendation. The system suggests items with a positive polarity term to the disabled user. The intelligent model and merged ontology were designed by employing Java and Protege Web Ontology Language 2 software, respectively. Experimentation results show that the proposed system is highly productive when analyzing retrieved information, and provides accurate recommendations.

Improving a full-text search engine: the importance of negation detection and family history context to identify cases in a biomedical data warehouse

The repurposing of electronic health records (EHRs) can improve clinical and genetic research for rare diseases. However, significant information in rare disease EHRs is embedded in the narrative reports, which contain many negated clinical signs and family medical history. This paper presents a method to detect family history and negation in narrative reports and evaluates its impact on selecting populations from a clinical data warehouse (CDW). We developed a pipeline to process 1.6 million reports from multiple sources. This pipeline is part of the load process of the Necker Hospital CDW. We identified patients with "Lupus and diarrhea," "Crohn's and diabetes," and "NPHP1" from the CDW. The overall precision, recall, specificity, and F-measure were 0.85, 0.98, 0.93, and 0.91, respectively. The proposed method generates a highly accurate identification of cases from a CDW of rare disease EHRs.

Supporting information retrieval from electronic health records: A report of University of Michigan’s nine-year experience in developing and using the Electronic Medical Record Search Engine (EMERSE)

ObjectiveThis paper describes the University of Michigan’s nine-year experience in developing and using a full-text search engine designed to facilitate information retrieval (IR) from narrative documents stored in electronic health records (EHRs). The system, called the Electronic Medical Record Search Engine (EMERSE), functions similar to Google but is equipped with special functionalities for handling challenges unique to retrieving information from medical text. Materials and methodsKey features that distinguish EMERSE from general-purpose search engines are discussed, with an emphasis on functions crucial to (1) improving medical IR performance and (2) assuring search quality and results consistency regardless of users’ medical background, stage of training, or level of technical expertise. ResultsSince its initial deployment, EMERSE has been enthusiastically embraced by clinicians, administrators, and clinical and translational researchers. To date, the system has been used in supporting more than 750 research projects yielding 80 peer-reviewed publications. In several evaluation studies, EMERSE demonstrated very high levels of sensitivity and specificity in addition to greatly improved chart review efficiency. DiscussionIncreased availability of electronic data in healthcare does not automatically warrant increased availability of information. The success of EMERSE at our institution illustrates that free-text EHR search engines can be a valuable tool to help practitioners and researchers retrieve information from EHRs more effectively and efficiently, enabling critical tasks such as patient case synthesis and research data abstraction. ConclusionEMERSE, available free of charge for academic use, represents a state-of-the-art medical IR tool with proven effectiveness and user acceptance.