Search Engine System Research Articles

Harnessing AI for Enhanced Searching in Digital Libraries: Transforming Research Practices

The application of Artificial Intelligence (AI) technologies in digital libraries is changing the paradigm in which users search and interact with digital content. This article is focused on how the use of AI technologies improves the search in digital libraries, demonstrating the benefits of such technologies in overcoming the challenges associated with formulation of query, information retrieval, and end user usage. Most search engine systems face the problems of user queries, which are inherent in online searches. Many user queries are vague, imprecise or ambiguous with regards to the context and as a result lead to less than satisfactory search results. AI technologies such as Natural Language Processing (NLP) and Machine Learning (ML) solve this challenge effectively as they enhance comprehension of queries, contextual understanding, and relevance of the results returned. With the capabilities of NLP incorporated in search engines, search queries which are in the form of natural language can be understood and executed which leads to efficiency in targeting the information needed. Comprehensive restructuring of standard information systems is used by computer learning algorithms for searches by inference and instills usage in users. This study also examines the place of AI in query formulation and the different ways it can be employed; fully automated, semi-automated and/or manually so that the users are less faced with the challenges that are usually present during the query formulation process. Such increases in the level of difficulty and scope of information retrieval suggest that search AI improved the scenario.

SSD In-Storage Computing for Search Engines

SSD-based in-storage computing (called ”Smart SSDs”) allows application-specific codes to execute inside SSDs to exploit the high internal bandwidth and energy-efficient processors. As a result, Smart SSDs have been successfully deployed in many industry settings, e.g., Samsung, IBM, Teradata, and Oracle. Moreover, researchers have also demonstrated their potential opportunities in database systems, data mining, and big data processing. However, it remains unknown whether search engine systems can benefit from Smart SSDs. This work takes a first step to answer this question. The major research issue is what search engine query processing operations can be cost-effectively offloaded to SSDs. For this, we carefully identified the five most commonly used search engine operations that could potentially benefit from Smart SSDs: intersection, ranked intersection, ranked union, difference, and ranked difference. With close collaboration with Samsung, we offloaded the above five operations of Apache Lucene (a widely used open-source search engine) to Samsungs Smart SSD. Finally, we conducted extensive experiments to evaluate the system performance and tradeoffs by using both synthetic datasets and real datasets. The experimental results show that Smart SSDs significantly reduce the query latency by a factor of 2-3 and energy consumption by 6-10 for most of the aforementioned operations.

A Longitudinal and Comprehensive Measurement of DNS Strict Privacy

The DNS privacy protection mechanisms, DNS over TLS (DoT) and DNS over HTTPS (DoH), only work correctly if both the server and client support the Strict Privacy profile and no vulnerability exists in the implemented TLS/HTTPS. A natural question then arises: what is the landscape of DNS Strict Privacy? To this end, we provide the first longitudinal and comprehensive measurement of DoT/DoH deployments in recursive resolvers, authoritative servers, and browsers. With the collected data, we find the number of DoT/DoH servers increased substantially during our ten-month-long scan. However, around 60% of DoT and 44% of DoH recursive resolver certificates are invalid. Worryingly, our measurements confirm the centralization problem of DoT/DoH. Furthermore, we classify DNS Strict Privacy servers into four levels according to daily scanning results on TLS/HTTPS-related security features. Unfortunately, around 25% of DoH Strict Privacy recursive resolvers fail to meet the minimum level requirements. To help the Internet community better perceive the landscape of DNS Strict Privacy, we implement a DoT/DoH server search engine and recommender system. Additionally, we investigate five popular browsers across four operating systems and find some inconsistent behavior with their DNS privacy implementations. For example, Firefox in Windows, Linux, and Android allows DoH communication with the server without the SAN certificate. At last, we advocate that all participants head together for a bright DNS Strict Privacy landscape by discussing current hindrances and controversies in DNS privacy.

Metadata Scraping Using Programmable Customized Search Engine

The World Wide Web (WWW) is a vast repository of knowledge, including intellectual, social, financial, and security-related data. Online information is typically accessed for instructional purposes. On the internet, information is accessible in a variety of formats and access interfaces. Because of this, indexing or semantic processing of the data via websites may be difficult. The method that seeks to resolve this issue is web data scraping. Unstructured web data can be converted into structured data using web data scraping so that it can be stored and examined in a central local database or spreadsheet. This paper offers a metadata scraping using a programmable Customized Search Engine (CSE) system, which can extract metadata from web pages (HTML pages) in the Google database and save it in an XML format for later analysis and retrieval. Documents that contain metadata are a relatively recent phenomenon on the web and increase the likelihood that users will find the information they need. Index Terms— Programmable (CSE), JSON API, API key, metadata scraping.

An Efficient and Robust Semantic Hashing Framework for Similar Text Search

Similar text search aims to find texts relevant to a given query from a database, which is fundamental in many information retrieval applications, such as question search and exercise search. Since millions of texts always exist behind practical search engine systems, a well-developed text search system usually consists of recall and ranking stages. Specifically, the recall stage serves as the basis in the system, where the main purpose is to find a small set of relevant candidates accurately and efficiently. Towards this goal, deep semantic hashing, which projects original texts into compact hash codes, can support good search performance. However, learning desired textual hash codes is extremely difficult due to the following problems. First, compact hash codes (with short length) can improve retrieval efficiency, but the demand for learning compact hash codes cannot guarantee accuracy due to severe information loss. Second, existing methods always learn the unevenly distributed codes in the space from a local perspective, leading to unsatisfactory code-balance results. Third, a large fraction of textual data contains various types of noise in real-world applications, which causes the deviation of semantics in hash codes. To this end, in this paper, we first propose a general unsupervised encoder-decoder semantic hashing framework, namely MASH (short for Memory-bAsed Semantic Hashing), to learn the balanced and compact hash codes for similar text search. Specifically, with a target of retaining semantic information as much as possible, the encoder introduces a novel relevance constraint among informative high-dimensional representations to guide the compact hash code learning. Then, we design an external memory where the hashing learning can be optimized in the global space to ensure the code balance of the learning results, which can promote search efficiency. Besides, to alleviate the performance degradation problem of the model caused by text noise, we propose an improved SMASH (short for denoiSing Memory-bAsed Semantic Hashing) model by incorporating a noise-aware encoder-decoder framework. This framework considers the noise degree for each text from the semantic deviation aspect, ensuring the robustness of hash codes. Finally, we conduct extensive experiments in three real-world datasets. The experimental results clearly demonstrate the effectiveness and efficiency of MASH and SMASH in generating balanced and compact hash codes, as well as the superior denoising ability of SMASH.

Pre-trained Language Model-based Retrieval and Ranking for Web Search

Pre-trained language representation models (PLMs) such as BERT and Enhanced Representation through kNowledge IntEgration (ERNIE) have been integral to achieving recent improvements on various downstream tasks, including information retrieval. However, it is nontrivial to directly utilize these models for the large-scale web search due to the following challenging issues: (1) the prohibitively expensive computations of massive neural PLMs, especially for long texts in the web document, prohibit their deployments in the web search system that demands extremely low latency; (2) the discrepancy between existing task-agnostic pre-training objectives and the ad hoc retrieval scenarios that demand comprehensive relevance modeling is another main barrier for improving the online retrieval and ranking effectiveness; and (3) to create a significant impact on real-world applications, it also calls for practical solutions to seamlessly interweave the resultant PLM and other components into a cooperative system to serve web-scale data. Accordingly, we contribute a series of successfully applied techniques in tackling these exposed issues in this work when deploying the state-of-the-art Chinese pre-trained language model, i.e., ERNIE, in the online search engine system. We first present novel practices to perform expressive PLM-based semantic retrieval with a flexible poly-interaction scheme and cost-efficiently contextualize and rank web documents with a cheap yet powerful Pyramid-ERNIE architecture. We then endow innovative pre-training and fine-tuning paradigms to explicitly incentivize the query-document relevance modeling in PLM-based retrieval and ranking with the large-scale noisy and biased post-click behavioral data. We also introduce a series of effective strategies to seamlessly interwoven the designed PLM-based models with other conventional components into a cooperative system. Extensive offline and online experimental results show that our proposed techniques are crucial to achieving more effective search performance. We also provide a thorough analysis of our methodology and experimental results.

User intent prediction search engine system based on query analysis and image recognition technologies

User intent prediction search engine system based on query analysis and image recognition technologies

Construction of Intelligent Search Engine for Big Data Multimedia Resource Subjects Based on Partial Least Squares Structural Equation

Abstract The equivalence of the research model is used as part of least squares to construct an intelligent search engine for large multimedia files. First of all, based on the intelligent design of big data analysis, optimize the search engine mode and create a good search environment. According to the research on big data technology, the application value of big data in intelligent search engine is analyzed, and a better search engine system is built on this basis. The basic theory of comparative modeling is also investigated. Then, on the big data management platform of Siping Fire Bureau, the operation of the big data search engine is identified according to the difference of evolutionary algorithms. Experiments show that the big data search engine, as part of the least squares structural equation algorithm, also improves user satisfaction, which is considered below. From the big data search engine, according to the difference of the evolutionary algorithm, the return time, return value and cost are improved, and the search itself can be effectively and accurately realized.

Test model for database architectures: an assessment for job search engine systems

Data is a fundamental part of information and management systems; thus, controlling its complexity is an essential step nowadays. NoSQL databases adopt new approaches to data management differing from relational structures. For this study, two database systems are considered, MongoDB as a NoSQL data storage model and PostgreSQL as a relational data model, they are compared and evaluated on a job search system. For this purpose, a dataset was defined, and its representation was constructed in databases based on each technology. This process allowed the data modeling in terms of the best practices, then, the development of a test plan prepared the environment for the determination of the comparison metrics of both databases under the methodology specified by the International Software Testing Qualifications Board (ISTQB) and the types of database testing. This study determined that the SQL schema provides greater functionality, that ensures the support of transactions and data integrity, the opposite happened with the NoSQL schemas, resulting in more efficient but lacking functionalities that are characteristic and required for data representation of a consistent system.

SybilHunter: Hybrid graph-based sybil detection by aggregating user behaviors

Nowadays, online social networks (OSNs) support users in sharing their information and providing valuable data for various applications, e.g., rating and recommendation systems, search engine systems, etc. In such online social network-based applications, information quality is essential. However, the easy-to-use interactive user interfaces and free publication mechanisms facilitate malicious users to launch Sybil Attacks. They create fake identities, pollute user-generated content, and cause server information quality problems, such as buzz, rumor, spam, etc. Most existing graph-based sybil detection approaches only consider the static graph structure features and heavily rely on the prior knowledge of labeled nodes. “Limited-attack-edge” assumption, the critical assumption of these approaches has been proved invalid in many scenarios.In this paper, we propose SybilHunter, a hybrid graph-based sybil detection approach by aggregating user social behavior patterns. Our approach refines the OSN structure, quantifies nodes’ similarity according to the dynamic user behavior features to evaluate user pairs’ trustworthiness and consistency. Then it constructs a weighted-strong-social (WSS) graph, based on which SybilHunter outputs sybil nodes. We simulate and evaluate our approach under a Weibo dataset. The AUC of SybilHunter achieves 0.945, which is significantly higher than typical graph-based sybil detection methods, such as SybilRank, SybilWalk, etc.

Epidemiological Study of Lung Cancer and Clinical Medication in England from 2001 to 2019.

We aimed to explore the epidemiological characteristics and changes of lung cancer and the clinical medication in England from 2001 to 2019. We searched related research using search engine systems such as MEDLINE, PubMed, and PsychINFO. Lung cancer is a serious disease and the prognosis is usually very poor. The overall mortality rate of lung cancer decreased year by year in England from 2001 to 2019, but men, the elderly, and people exposed to polluted air are still more likely to be infected with lung cancer or die as a result, the prevalence and mortality rate of lung cancer in the north of England is significantly higher than that in the south, and the gap is increasing year by year. Lung cancer has changeable risk factors such as quitting smoking and improving air quality, which can effectively reduce the related risk. Paclitaxel, docetaxel, gemcitabine, and vinorelbine are the main drugs for the treatment of lung cancer in England and the treatment of these drugs is beneficial to the survival and quality of life of patients. Men and the elderly are at high risk of lung cancer, which means that lung cancer has obvious gender inequality and age inequality. At the same time, based on the statistical data of lung cancer risk in different regions, it can be concluded that lung cancer also has strong geographical and economic inequality. Changing risk factors and using drugs can effectively reduce the risk of lung cancer and provide effective treatment.

Deep Learning Assist IoT Search Engine for Disaster Damage Assessment

ABSTRACT In this paper, we address the issue of disaster damage assessments using deep learning (DL) techniques. Specifically, we propose integrating DL techniques into the Internet of Things Search Engine (IoTSE) system to carry out disaster damage assessment. Our approach is to design two scenarios, Single and Complex Event Settings, to complete performance validation using four Convolutional Neural Network (CNN) models. These two scenarios are designed with three possible network services. Our experimental results confirm that all four CNN models can learn each label during the single event setting well. Whereas, with complex event settings, the CNN models have learning difficulty because multiple events have closely related labels.

Improving the usability of faceted exploratory search tools

For some years now, the faceted exploratory search has faced tremendous challenges to devise a simple, instantly understandable, and usable user interface to explore the vast amount of data and information available on the Internet. In our previous research, we have indicated certain burdens with some of the existing approaches. Therefore, in this research, we are trying to uncover the attributes that influence users to use the faceted feature of the exploratory search technique. In this paper, we presented several key elements of existing search engine systems, the best of their major features were explored, and guidelines that will help to improve the usability of faceted exploratory search have been suggested and offers a further understanding of the fundamental characteristics of the faceted search, which shows the motivations to use the search, also presented the main parts of a pluggable search system.

Knuth Morris Pratt String Matching Algorithm in Searching for Zakat Information and Social Activities

Algorithms are one of the components that need to be considered in the development of information systems. Determination of the algorithm is adjusted to the purpose of the system to be built. One algorithm that can be used is string matching. The string matching algorithm will play a role in searching for a string consisting of several characters (usually called a pattern). The method used in this research is string matching knuth morris pratt (KMP) which is used to search zakat information and social activities in the search engine system. KMP is a string matching algorithm with good performance. The results showed the performance of string matching using the KMP algorithm with 5 trials of input pattern on zakat information with execution times of 0.03 ms, 0.03 ms, 0.02 ms, 0.02 ms and 0.03 ms. And 5 times the input pattern experiment on social activities with execution time of 0.02 ms, 0.02 ms, 0.03 ms, 0.03 ms and 0.02 ms. Thus the average execution time of the KMP algorithm in string matching is 0.026 ms and 0.024 ms.

Improved FC-LFM Algorithm Integrating Time Decay Factor

Different from the search engine system, the recommendation system can help users quickly find their interesting items from the massive data in a personalized way. Traditional collaborative filtering algorithms based on users and items need to define the granularity, dimension, and weight of classification subjectively when calculating similarity based on user behavior. So the accuracy and calculation efficiency of prediction scoring results are not high enough. LFM (latent factor model) based on data itself, adopts automatic clustering according to user behavior and uses a machine-learning method to mine hidden features from the user’s historical scoring data. But when the amount of data is large, there exists data sparsity in the user rating matrix. Also, the user’s interest is always changing with time, and the items themselves have a certain life cycle. Based on the above problems, this paper first proves that the accuracy of LFM model is influenced by the popularity and diversity of negative samples through comparative experiments. Then, a new algorithm called FC-LFM (forgetting curve-latent factor model) is proposed. In this algorithm, the Ebbinghaus forgetting curve function is introduced to improve LFM model and the time decay factor is integrated into the iterative operation of negative sample popularity, matrix filling, user feature matrix, and item feature matrix. In the end, the improved FC-LFM collaborative filtering algorithm is proved to be superior to the traditional UserCF, UserCF-IIF, ItemCF, ItemCF-IUF, and LFM algorithm in accuracy and recall rate by comparing experiments on the MovieLens data set.

A Framework (SOCRATex) for Hierarchical Annotation of Unstructured Electronic Health Records and Integration Into a Standardized Medical Database: Development and Usability Study.

BackgroundAlthough electronic health records (EHRs) have been widely used in secondary assessments, clinical documents are relatively less utilized owing to the lack of standardized clinical text frameworks across different institutions.ObjectiveThis study aimed to develop a framework for processing unstructured clinical documents of EHRs and integration with standardized structured data.MethodsWe developed a framework known as Staged Optimization of Curation, Regularization, and Annotation of clinical text (SOCRATex). SOCRATex has the following four aspects: (1) extracting clinical notes for the target population and preprocessing the data, (2) defining the annotation schema with a hierarchical structure, (3) performing document-level hierarchical annotation using the annotation schema, and (4) indexing annotations for a search engine system. To test the usability of the proposed framework, proof-of-concept studies were performed on EHRs. We defined three distinctive patient groups and extracted their clinical documents (ie, pathology reports, radiology reports, and admission notes). The documents were annotated and integrated into the Observational Medical Outcomes Partnership (OMOP)-common data model (CDM) database. The annotations were used for creating Cox proportional hazard models with different settings of clinical analyses to measure (1) all-cause mortality, (2) thyroid cancer recurrence, and (3) 30-day hospital readmission.ResultsOverall, 1055 clinical documents of 953 patients were extracted and annotated using the defined annotation schemas. The generated annotations were indexed into an unstructured textual data repository. Using the annotations of pathology reports, we identified that node metastasis and lymphovascular tumor invasion were associated with all-cause mortality among colon and rectum cancer patients (both P=.02). The other analyses involving measuring thyroid cancer recurrence using radiology reports and 30-day hospital readmission using admission notes in depressive disorder patients also showed results consistent with previous findings.ConclusionsWe propose a framework for hierarchical annotation of textual data and integration into a standardized OMOP-CDM medical database. The proof-of-concept studies demonstrated that our framework can effectively process and integrate diverse clinical documents with standardized structured data for clinical research.

A difference-of-convex programming approach with parallel branch-and-bound for sentence compression via a hybrid extractive model

Sentence compression is an important problem in natural language processing with wide applications in text summarization, search engine and human-AI interaction system etc. In this paper, we design a hybrid extractive sentence compression model combining a probability language model and a parse tree language model for compressing sentences by guaranteeing the syntax correctness of the compression results. Our compression model is formulated as an integer linear programming problem, which can be rewritten as a Difference-of-Convex (DC) programming problem based on the exact penalty technique. We use a well-known efficient DC algorithm -- DCA to handle the penalized problem for local optimal solutions. Then a hybrid global optimization algorithm combining DCA with a parallel branch-and-bound framework, namely PDCABB, is used for finding global optimal solutions. Numerical results demonstrate that our sentence compression model can provide excellent compression results evaluated by F-score, and indicate that PDCABB is a promising algorithm for solving our sentence compression model.

Web Crawler: Design And Implementation For Extracting Article-Like Contents

The World Wide Web is a large, wealthy, and accessible information system whose users are increasing rapidly nowadays. To retrieve information from the web as per users’ requests, search engines are built to access web pages. As search engine systems play a significant role in cybernetics, telecommunication, and physics, many efforts were made to enhance their capacity.However, most of the data contained on the web are unmanaged, making it impossible to access the entire network at once by current search engine system mechanisms. Web Crawler, therefore, is a critical part of search engines to navigate and download full texts of the web pages. Web crawlers may also be applied to detect missing links and for community detection in complex networks and cybernetic systems. However, template-based crawling techniques could not handle the layout diversity of objects from web pages. In this paper, a web crawler module was designed and implemented, attempted to extract article-like contents from 495 websites. It uses a machine learning approach with visual cues, trivial HTML, and text-based features to filter out clutters. The outcomes are promising for extracting article-like contents from websites, contributing to the search engine systems development and future research gears towards proposing higher performance systems.

System Design of Cloud Search Engine Based on Rich Text Content

In order to improve the search performance of rich text content, a cloud search engine system based on rich text content is designed. On the basis of traditional search engine hardware system, several hardware devices such as Solr index server, collector, Chinese word segmentation device and searcher are installed, and the data interface is adjusted. On the basis of hardware equipment and database support, this paper uses the open source Apache Tika framework to obtain the metadata of rich text documents, implements word segmentation according to the rich text content and semantics, and calculates the weight of each keyword. Input search keywords, establish a text index, use BM25 algorithm to calculate the similarity between keywords and text, and output the search results of rich text according to the similarity calculation results. The experimental results show that the design system has high recall rate, high throughput, and the construction time of each data item index in different files is short, which improves the search efficiency and search accuracy.

A survey on network node ranking algorithms: Representative methods, extensions, and applications

The ranking of network node importance is one of the most essential problems in the field of network science. Node ranking algorithms serve as an essential part in many application scenarios such as search engine, social networks, and recommendation systems. This paper presents a systematic review on three representative methods: node ranking based on centralities, PageRank algorithm, and HITS algorithm. Furthermore, we investigate the latest extensions and improvements of these representative methods, provided with several main application fields. Inspired by the survey of current literatures, we attempt to propose promising directions for future research. The conclusions of this paper are enlightening and beneficial to both the academic and industrial communities.