The System Architecture for the Basic Information of Science and Technology Experts Based on Distributed Storage and Web Mining

Abstract

In order to build an efficient basic information system of science and technology experts based on Web mining, a novel system architecture for application is proposed in this paper. The proposed system architecture integrates spider module, local distributed storage and Mongo-DB. The basic experts information of science and technology appeared in the Websites are synthesized as two format and using two strategies to deal with it respectively. The normalized texts which extracted from Web page by URLs are suggested. The extracted results include the name, sex, birth, hometown and professional title of science and technology experts respectively. The data stream flow, the information management model for the users and science and technology experts, the target website URLs and URLs management model, and data processing module are introduced in detailed. The synchronization of multiple databases and replica sets architecture for sharing cluster architecture is proposed in application system. Experiments show that the application system obtains a very high efficiency. The results show as by proposed system architecture can satisfy the application requirements for the customer.