Domain-specific Search Research Articles

CURRENT STATUS OF THE BENCHMARK DATABASE BEMEDA

Abstract. Open science is an important attribute for developing new approaches. Especially, the data component plays a significant role. The FAIR principle provides a good orientation towards open data. One part of FAIR is findability. Thus, domain specific dataset search platforms were developed: the Earth Observation Database and our Benchmark Metadata Database (BeMeDa). In addition to the search itself, the datasets found by this platforms can be compared with each other with regard to their interoperability. We compare these two platforms and present an update of our platform BeMeDa. This update includes additional location information about the datasets and a new frontend design with improved usability. We rely on user feedback for further improvements and enhancements.

Heuristic Search for Physics-Based Problems: Angry Birds in PDDL+

This paper studies how a domain-independent planner and combinatorial search can be employed to play AngryBirds, a well established AI challenge problem. To model the game, we use PDDL+, a planning language for mixed discrete/continuous domains that supports durative processes and exogenous events. The paper describes the PDDL+ model and identifies key design decisions that reduce the problem complexity. In addition, we propose several domain-specific enhancements including heuristics and a search technique similar to preferred operators. Together, they alleviate the complexity of combinatorial search. We evaluate our approach by comparing its performance with dedicated domain-specific solvers on a range of Angry Birds levels. The results show that our performance is on par with these domain-specific approaches in most levels, even without using our domain-specific search enhancements.

Improving the Effectiveness and Efficiency of Web-Based Search Tasks for Policy Workers

We adapt previous literature on search tasks for developing a domain-specific search engine that supports the search tasks of policy workers. To characterise the search tasks we conducted two rounds of interviews with policy workers at the municipality of Utrecht, and found that they face different challenges depending on the complexity of the task. During simple tasks, policy workers face information overload and time pressures, especially during web-based searches. For complex tasks, users prefer finding domain experts within their organisation to obtain the necessary information, which requires a different type of search functionality. To support simple tasks, we developed a web search engine that indexes web pages from authoritative sources only. We tested the hypothesis that users prefer expert search over web search for complex tasks and found that supporting complex tasks requires integrating functionality that enables finding internal experts within the broader web search engine. We constructed representative tasks to evaluate the proposed system’s effectiveness and efficiency, and found that it improved user performance. The search functionality developed could be standardised for use by policy workers in different municipalities within the Netherlands.

Searching a High Performance Feature Extractor for Text Recognition Network.

Feature extractor plays a critical role in text recognition (TR), but customizing its architecture is relatively less explored due to expensive manual tweaking. In this article, inspired by the success of neural architecture search (NAS), we propose to search for suitable feature extractors. We design a domain-specific search space by exploring principles for having good feature extractors. The space includes a 3D-structured space for the spatial model and a transformed-based space for the sequential model. As the space is huge and complexly structured, no existing NAS algorithms can be applied. We propose a two-stage algorithm to effectively search in the space. In the first stage, we cut the space into several blocks and progressively train each block with the help of an auxiliary head. We introduce the latency constrain into the second stage and search sub-network from the trained supernet via natural gradient descent. In experiments, a series of ablation studies are performed to better understand the designed space, search algorithm, and searched architectures. We also compare the proposed method with various state-of-the-art ones on both hand-written and scene TR tasks. Extensive results show that our approach can achieve better recognition performance with less latency. Code is avaliable at https://github.com/AutoML-Research/TREFE.

Learning Classical Planning Strategies with Policy Gradient

A common paradigm in classical planning is heuristic forward search. Forward search planners often rely on simple best-first search which remains fixed throughout the search process. In this paper, we introduce a novel search framework capable of alternating between several forward search approaches while solving a particular planning problem. Selection of the approach is performed using a trainable stochastic policy, mapping the state of the search to a probability distribution over the approaches. This enables using policy gradient to learn search strategies tailored to a specific distributions of planning problems and a selected performance metric, e.g. the IPC score. We instantiate the framework by constructing a policy space consisting of five search approaches and a two-dimensional representation of the planner’s state. Then, we train the system on randomly generated problems from five IPC domains using three different performance metrics. Our experimental results show that the learner is able to discover domain-specific search strategies, improving the planner’s performance relative to the baselines of plain bestfirst search and a uniform policy.

A meta-engine for building domain-specific search engines

<h2>Abstract</h2> In recent years, domain-specific search (DSS) has emerged as a growing and important area of applied research in artificial intelligence (AI) and information retrieval (IR). Over the last 6 years of research, our group has developed a ‘meta-engine' called myDIG (my Domain-specific Insight Graphs) that provides a relatively easy and customizable workflow for building DSSs without advanced technical training in crawling, information retrieval or user-interfaces. The myDIG system has been applied to some important and difficult use cases (most notably, fighting human trafficking), in addition to being used in classrooms by graduate students for building complex DSSs from scratch.

A Domain-Specific Search Engine: A Case of University of Abuja

Users need more effective ways of organizing and searching for information. It has become increasingly difficult for users to findinformation on the World Wide Web that satisfies their individual needs since information resources on the World Wide Web continue to grow. The reason is that the World Wide Web provides a bunch of information that is generic in perspective. Although, the generic search engine makes attempts to streamline searching to search-key by traversing and combing the whole bunch of stored information. Under these circumstances, the system turns out to be inefficient and many times the results do not tally to what users desire. This paper presents a system to find interesting textual content among tons of documents taking the department of computer science of the University of Abuja, Nigeria as a case study. The paper proposed a vector space ranking algorithm which is a content-based ranking method that allows the user to utilize the full co-occurrence matrix of all words in the corpus to bring out relevant material using a simple and structured query interface. The index structures of the system have been specifically designed to support the ranking scheme. One important aspect of the system is its flexibility and robustness that makes it adaptable to any domain rather than being tailored to a particular one.

Design of a vertical search engine for synchrotron data: a big data approach using Hadoop ecosystem

A synchrotron as an experimental physics facility can provide the opportunity of a multi-disciplinary research and collaboration between scientists in various fields of study such as physics, chemistry, etc. During the construction and operation of such facility valuable data regarding the design of the facility, instruments and conducted experiments are published and stored. It takes researchers a long time going through different results from generalized search engines to find their needed scientific information so that the design of a domain specific search engine can help researchers to find their desired information with greater precision. It also provides the opportunity to use the crawled data to create a knowledgebase and also to generate different datasets required by the researchers. There have been several other vertical search engines that are designed for scientific data search such as medical information. In this paper we propose the design of such search engine on top of the Apache Hadoop framework. Usage of Hadoop ecosystem provides the necessary features such as scalability, fault tolerance and availability. It also abstracts the complexities of search engine design by using different open source tools as building blocks, among them Apache Nutch for the crawling block and Apache Solr for indexing and query processing. Our primary results obtained by implementing the proposed method in single node mode, the index of over a hundred thousand pages was created with the average fetch interval of 30 days having 28 segments and approximately 570 MB size. The performance factors such as the usage of available bandwidth and system load were logged using Linux’s sysstat package.

Similarity Driven Unsupervised Learning for Materials Science Terminology Extraction

Knowledge of material properties, microstructure, underlying material composition and manufacturing process parameters that the material has undergone is of significant interest to materials scientists and engineers. A large amount of information of this nature is present in the form of unstructured sources. To access the right information for a given problem at hand, various domain specific search systems have been developed. Domain terminologies, when available, can significantly improve the quality of such systems. In this paper, we propose a novel similarity driven learning approach for automatic terminology extraction for materials science domain. It first uses various intra-domain and inter-domain unsupervised corpus level features to score and rank candidate terminologies. For inter-domain features, we use British National Corpus (BNC) as the general purpose corpus. The ranked candidate terms are then used to generate training data for learning a similarity based scoring function. The parameters of this scoring function are learnt using a Siamese neural network which uses word embeddings learnt from both the domain as well as the general purpose corpora to leverage contrasting term features. The proposed similarity based learning approach consistently outperforms other reported classification approaches on the materials dataset.

An Approach for Focused Crawler to Harvest Digital Academic Documents in Online Digital Libraries

With the rapid growth of digital information and user need, it becomes imperative to retrieve relevant and desired domain or topic specific documents as per the user query quickly. A focused crawler plays a vital role in digital libraries to crawl the web so that researchers can easily explore the domain specific search results list and find the desired content against the query. In this article, a focused crawler is being proposed for online digital library search engines, which considers meta-data of the query in order to retrieve the corresponding document or other relevant but missing information (e.g. paid publication from ACM, IEEE, etc.) against the user query. The different query strategies are made by using the meta-data and submitted to different search engines which aim to find more relevant information which is missing. The result comes out from these search engines are filtered and then used further for crawling the Web.

Constructing Domain-Specific Search Engines With No Programming

We propose a demonstration of myDIG (my Domain-specific Insight Graphs), a system that allows non-technical domain experts, including those with no programming experience, to construct a domain-specific search engine over a raw corpus of webpages. myDIG has been developed and refined over multiple years under the DARPA MEMEX program, and has undergone rigorous user testing with actual domain experts from investigative agencies like the Securities and Exchange Commission (SEC). All components of myDIG are open-source, and the product of fundamental research.

Searching in the Context of a Task: A Review of Methods and Tools

Contextual information extracted from the user task can help to better target retrieval to task-relevant content. In particular, topical context can be exploited to identify the subject of the information needs, contributing to reduce the information overload problem. A great number of methods exist to extract raw context data and contextual interaction patterns from the user task and to model this information using higher-level representations. Context can then be used as a source for automatic query generation, or as a means to refine or disambiguate user-generated queries. It can also be used to filter and rank results as well as to select domain-specific search engines with better capabilities to satisfy specific information requests. This article reviews methods that have been applied to deal with the problem of reflecting the current and long-term interests of a user in the search process. It discusses major difficulties encountered in the research area of context-based information retrieval and presents an overview of tools proposed since the mid-nineties to deal with the problem of context-based search.

A Relation Aware Search Engine for Materials Science

Knowledge of material properties, microstructure, underlying material composition, and manufacturing process parameters that the material has undergone is of significant interest to materials scientists and engineers. A large amount of information of this nature is available in publications in the form of experimental measurements, simulation results, etc. However, getting to the right information of this kind that is relevant for a given problem on hand is a non-trivial task. First, an engineer has to go through a large collection of documents to select the right ones. Then, the engineer has to scan through these selected documents to extract relevant pieces of information. Our goal is to help automate some of these steps. Traditional search engines are not of much help here, as they are keyword centric and weak on relation processing. In this paper, we present a domain-specific search engine that processes relations to significantly improve search accuracy. The engine preprocesses material publication repositories to extract entities such as material compositions, material properties, manufacturing processes, process parameters, and their values and builds an index using these entities and values. The engine then uses this index to process user queries to retrieve relevant publication fragments. It provides a domain-specific query language with relational and logical operators to compose complex queries. We have conducted an experiment on a small library of publications on steel on which searches such as “get the list of publications which have carbon composition between 0.2 and 0.3 and on which tempering is carried out for about 30 to 40 min” are performed. We compare the results of our search engine with the results of a keyword-based search engine.

Granular information retrieval using neighborhood systems

With the rapid growth of the amount of information stored on networks such as the internet, it is more difficult for information seekers to retrieve relevant information. This paper illustrates the design and improvement of a near neighborhood approach of information retrieval system to facilitate domain specific search. In exacting, a novel model depending on the notion of neighborhood system designed to rank documents according the searchers specific granularity requirements. The initial experiments confirm that our approach outperforms a classical vector‐based information retrieval system. Our research work opens the door to the design and development of the next generation of internet search engines to alleviate the problem of information overload using more topological concepts.

What do developers search for on the web?

Developers commonly make use of a web search engine such as Google to locate online resources to improve their productivity. A better understanding of what developers search for could help us understand their behaviors and the problems that they meet during the software development process. Unfortunately, we have a limited understanding of what developers frequently search for and of the search tasks that they often find challenging. To address this gap, we collected search queries from 60 developers, surveyed 235 software engineers from more than 21 countries across five continents. In particular, we asked our survey participants to rate the frequency and difficulty of 34 search tasks which are grouped along the following seven dimensions: general search, debugging and bug fixing, programming, third party code reuse, tools, database, and testing. We find that searching for explanations for unknown terminologies, explanations for exceptions/error messages (e.g., HTTP 404), reusable code snippets, solutions to common programming bugs, and suitable third-party libraries/services are the most frequent search tasks that developers perform, while searching for solutions to performance bugs, solutions to multi-threading bugs, public datasets to test newly developed algorithms or systems, reusable code snippets, best industrial practices, database optimization solutions, solutions to security bugs, and solutions to software configuration bugs are the most difficult search tasks that developers consider. Our study sheds light as to why practitioners often perform some of these tasks and why they find some of them to be challenging. We also discuss the implications of our findings to future research in several research areas, e.g., code search engines, domain-specific search engines, and automated generation and refinement of search queries.

Distributed agents for online spatial searches

As the availability and utilisation of online data blossoms, automated online searches—whether to answer a simple question, seek specific sensor readings, or investigate research in a particular domain—have raised a number of issues. Simple search tools do not access the deep web of services and online forms, and cannot handle knowledge domain-specific search problems, but specialist search tools can have a narrow domain and applicability. Some online tools circumvent these problems by putting more filter controls into the hands of users, but this leads to more complex interfaces which can raise usability barriers. A distributed approach, where specialised search agents act autonomously to find contextualised information, can provide a useful compromise between a simple, general search interface and specialist searches. This paper outlines work in progress on design and use of specialist search agents, with a case study to find public transportation bus stops within a spatial region. The approach is demonstrated with a proof of concept web interface, developed to interpret a text query to find and show bus stop locations within a named boundary by coordinating multiple online search agents. Search agents were designed to follow a common model to allow for future development of agent types, including specialist agents used in the case study to search standard open web services and extract spatial features.

BIOMedical Search Engine Framework: Lightweight and customized implementation of domain-specific biomedical search engines

Background and objectivesText mining and semantic analysis approaches can be applied to the construction of biomedical domain-specific search engines and provide an attractive alternative to create personalized and enhanced search experiences. Therefore, this work introduces the new open-source BIOMedical Search Engine Framework for the fast and lightweight development of domain-specific search engines. The rationale behind this framework is to incorporate core features typically available in search engine frameworks with flexible and extensible technologies to retrieve biomedical documents, annotate meaningful domain concepts, and develop highly customized Web search interfaces. MethodsThe BIOMedical Search Engine Framework integrates taggers for major biomedical concepts, such as diseases, drugs, genes, proteins, compounds and organisms, and enables the use of domain-specific controlled vocabulary. Technologies from the Typesafe Reactive Platform, the AngularJS JavaScript framework and the Bootstrap HTML/CSS framework support the customization of the domain-oriented search application. Moreover, the RESTful API of the BIOMedical Search Engine Framework allows the integration of the search engine into existing systems or a complete web interface personalization. ResultsThe construction of the Smart Drug Search is described as proof-of-concept of the BIOMedical Search Engine Framework. This public search engine catalogs scientific literature about antimicrobial resistance, microbial virulence and topics alike. The keyword-based queries of the users are transformed into concepts and search results are presented and ranked accordingly. The semantic graph view portraits all the concepts found in the results, and the researcher may look into the relevance of different concepts, the strength of direct relations, and non-trivial, indirect relations. The number of occurrences of the concept shows its importance to the query, and the frequency of concept co-occurrence is indicative of biological relations meaningful to that particular scope of research. Conversely, indirect concept associations, i.e. concepts related by other intermediary concepts, can be useful to integrate information from different studies and look into non-trivial relations. ConclusionsThe BIOMedical Search Engine Framework supports the development of domain-specific search engines. The key strengths of the framework are modularity and extensibilityin terms of software design, the use of open-source consolidated Web technologies, and the ability to integrate any number of biomedical text mining tools and information resources. Currently, the Smart Drug Search keeps over 1,186,000 documents, containing more than 11,854,000 annotations for 77,200 different concepts. The Smart Drug Search is publicly accessible at http://sing.ei.uvigo.es/sds/. The BIOMedical Search Engine Framework is freely available for non-commercial use at https://github.com/agjacome/biomsef.

Bound to Plan: Exploiting Classical Heuristics via Automatic Translations of Tail-Recursive HTN Problems

Hierarchical Task Network (HTN) planning is a formalism that can express constraints which cannot easily be expressed by classical (non-hierarchical) planning approaches. It enables reasoning about procedural structures and domain-specific search control knowledge. Yet the cornucopia of modern heuristic search techniques remains largely unincorporated in current HTN planners, in part because it is not clear how to estimate the goal distance for a partially-ordered task network. When using SHOP2-style progression, a task network of yet unprocessed tasks is maintained during search. In the general case it can grow arbitrarily large. However, many — if not most — existing HTN domains have a certain structure (called tail-recursive) where the network's size is bounded. We show how this bound can be calculated and exploited to automatically translate tail-recursive HTN problems into non-hierarchical STRIPS representations, which allows using both hierarchical structures and classical planning heuristics. In principle, the approach can also be applied to non-tail-recursive HTNs by incrementally increasing the bound. We give three translations with different advantages and present the results of an empirical evaluation with several HTN domains that are translated to PDDL and solved by two current classical planning systems. Our results show that we can automatically find practical bounds for solving partially-ordered HTN problems. We also show that classical planners perform similarly with our automatic translations versus a previous hand-bounded HTN translation which is restricted to totally-ordered problems.

Optimizing synthesis with metasketches

Many advanced programming tools---for both end-users and expert developers---rely on program synthesis to automatically generate implementations from high-level specifications. These tools often need to employ tricky, custom-built synthesis algorithms because they require synthesized programs to be not only correct, but also optimal with respect to a desired cost metric, such as program size. Finding these optimal solutions efficiently requires domain-specific search strategies, but existing synthesizers hard-code the strategy, making them difficult to reuse. This paper presents metasketches, a general framework for specifying and solving optimal synthesis problems. metasketches make the search strategy a part of the problem definition by specifying a fragmentation of the search space into an ordered set of classic sketches. We provide two cooperating search algorithms to effectively solve metasketches. A global optimizing search coordinates the activities of local searches, informing them of the costs of potentially-optimal solutions as they explore different regions of the candidate space in parallel. The local searches execute an incremental form of counterexample-guided inductive synthesis to incorporate information sent from the global search. We present Synapse, an implementation of these algorithms, and show that it effectively solves optimal synthesis problems with a variety of different cost functions. In addition, metasketches can be used to accelerate classic (non-optimal) synthesis by explicitly controlling the search strategy, and we show that Synapse solves classic synthesis problems that state-of-the-art tools cannot.

Multi Agent Architecture for Search Engine

The process of retrieving information is becoming ambiguous day by day due to huge collection of documents present on web. A single keyword produces millions of results related to given query but these results are not up to user expectations. The search results produced from traditional text search engines may be relevant or irrelevant. The underlying reason is Web documents are HTML documents that do not contain semantic descriptors and annotations. The paper proposes multi agent architecture to produce fewer but personalized results. The purpose of the research is to provide platform for domain specific personalized search. Personalized search allows delivering web pages in accordance with user’s interest and domain. The proposed architecture uses client side as well server side personalization to provide user with personalized fever but more accurate results. Multi agent search engine architecture uses the concept of semantic descriptors for acquiring knowledge about given domain and leading to personalized search results. Semantic descriptors are represented as network graph that holds relationship between given problem in form of hierarchy. This hierarchical classification is termed as Taxonomy.