Design and Implementation of Academic Search Aggregation Engine

Abstract

With the rapid development of the global Internet, academic search engines have become indispensable tools for academic research like literature, papers, books, preprints, and abstracts. The total amount of network information is growing rapidly today, and the global paper output doubles every five years. For academic researchers, it is very difficult to find papers that meet the needs in a large number of articles. Additionally, as the quality of journal papers varied, finding high-quality papers and accurately locating them is getting harder and harder. Many researchers find it tiring and inefficient that when looking for research fields, they need to read through a large number of articles in search results, and manually classify them. Through the clustering algorithm and data visualization and other related technologies, the systematic integration and classification of the search papers will help the researchers to accurately locate the required articles, and can greatly reduce the time for searching the documents. Regarding the previously stated problems, this thesis designs and implements an academic search aggregation engine (referred to as “aggregation engine”) based on cloud computing and some clustering algorithms. On the one hand, the aggregation engine distributes the user’s search request to multiple academic search engines, and then collects and summarizes the paper results, thereby uniformly displaying the search results; on the other hand, from the large number of papers obtained from academic search engines, the aggregation engine analyzes their titles, abstracts, and keywords, clusters papers with similar research points, and visualize clustering results to users, so that the result can be macroscopic and intuitive, grasping all the papers and the research points they involve. This paper first introduces the research background of the subject, through the investigation of the existing search engine and the existing aggregate search engine, it analyzes the needs of the researcher and other customers for the paper search; according to the demand analysis, we proposes a system design that distribute the search request to multiple academic search engines and then aggregated results. Finally, we realised the implementation of the aggregate search engine, and verified the effectiveness of the system through tests