Crawler Tool Research Articles

A Study on Unofficial Geographic Location Data Acquisition Technology Path under the Background of Big Data Era

The arrival of the era of big data has made people break through the bottleneck of various technologies and resources faced by traditional social sciences, but at the same time, new problems such as data acquisition, data value mining, data storage, circulation and exchange, data privacy and security have also brought new challenges to people. In order to explore a new methodology of geographic location data acquisition, this paper proposed the technical path of unofficial geographic location data acquisition by using web crawler tools and test the feasibility of it.

ANALYSING SENTIMENTS OF ONLINE REVIEWS ON RESTAURANTS IN MALAYSIA USING PREDICTIVE TEXT ANALYTICS

This study aims to analyse the overall sentiments of online reviews on restaurants in Malaysia using predictive text analytics. As we know in opinion mining, sentiment analysis is a prominent technique in predictive text mining. It is a technique that categorises opinions in unstructured text format into binary classification (ie. good or bad). The authors attempt to go beyond the binary classification by viewing texts as empirical entities derived using the Term Frequency - Inverse Document Frequency (TF-IDF) weighting algorithm. These empirical entities, based on online reviews of restaurants in Malaysia, are then manifested into hypothetically defined constructs closely reflecting their thematic and semantic nature. The were 4914 customer reviews from restaurants across 20 towns and cities in Malaysia scraped off TripAdvisor.com using web crawler tools. Then a series of analytical tests were carried out. First the online reviews were parsed, filtered and clustered using SAS Text Miner. Then the online reviews underwent the TF-IDF process to identify significant terms and weightages were assigned according to their importance. The TF-IDF process resulted in a series of important nouns and adjectives from the text corpus. Using these weightages of nouns and adjectives, the authors went on to thematise these terms based on their semantic nature to manifest hypothetical constructs. These constructs were based on the Mehrabian–Russell Stimulus Response Model. Subsequently the authors tested the associations among the constructs using variance-based and covariance-based Structural Equation Modelling (SEM). The authors were encouraged by this exploratory methodological approach in formulating predictive text analytics using SEM. Results indicated that sentiments were generally positive towards restaurants and the important terms derived were price, hospitality, location, waiting time, availability of parking and size of food portion.

A Method of Extracting The Semi-structured Data Implication Rules

The semi-structured data is a data form between structured data and unstructured data. Now it has been extensively used at the internet platforms and implied a lot of valuable information. In view of the semi-structured data implication rules, the paper discussed a method of extracting the semi-structured data implication rules, and designed the genetic algorithm SDIR of the semi-structured data implication rules based on the method of data conversion and genetic algorithm. Using network crawler tools to complete the semi-structured data collection and data conversion, the experiment verifies it’s effective in implication rules by crawling the data of “Dianping.com”.

AN IMAGE CRAWLER FOR CONTENT BASED IMAGE RETRIEVAL SYSTEM

Images from the minute it was invented, has had an immense impact on the world we live in. The extract ing the required images from the World Wide Web (WWW) is very difficult because web contains a huge number of images. To solve this problem we need a system that can retrieve the required images needed by the user. Image Crawler is a web based tool that colle cts and indexes group of web images available on the internet. This tool collect s the keyword or phrase from the user to retrieve tfrom the web. Then these collected keyword is applied to the different general search tools like Google, Yahoo, Bind etc, . The collected web page information is stored in the temporary file till 200KB file size from the server. Then this file cont ent will be scanned and extract the image URL’s and it is compared the URL which is pre sent in the database to avoid the duplicate downloa ds. The extracted URL’s images are downloaded and finally stores unique im age and corresponding metadata like filename, url, size etc. in the database. In this paper we present the designing of an Image cra wler tool. We build a search tool which is flexible , general-purpose image search framework and explore a directed result aggregating and removing of duplicates to achieve top results compared to other existing search tools. Finally this resulted images are used in the Content Based Image Retrieval (CBIR) system for extracting the relevant images need by the client using the content of the images rather than the text based information.

Pipeline Crawlers: Pigging the Unpiggable

This article, written by Assistant Technology Editor Karen Bybee, contains highlights of paper OTC 21210, ’Genesis Pipeline Crawlers: Pigging the Unpiggable,’ by Andy Kenney, Subsea Integrity, originally prepared for the 2011 Offshore Technology Conference, Houston, 2-5 May. The paper has not been peer reviewed. Many pipelines cannot be inspected internally because of reductions/changes in inside diameter (ID), extreme angles, 90º miter bends, T- and Y-junctions, or simply having access for pigs without a recovery point. A range of high-powered, tethered, brush-tractor crawlers are delivery systems for applications including internal inspection, cleaning, and repair. The brush-drive system copes with internal pipeline irregularities caused by corrosion, sedimentation, and changes in ID. Only one access point is needed for both insertion and recovery because crawlers can reverse. Introduction Conventional intelligent pigging of pipelines is usually a single-directional process (unless flow can be reversed) that requires sufficient product flow to drive the pig, a consistent bore size to allow the drive cups to efficiently occlude the pipeline, and a launch and receive trap at either end of the run. More than 50% of the global pipeline network (for various reasons) does not enable conventional pigging to be undertaken, and these pipelines are collectively identified under the heading of “unpiggable.” As ongoing development technology progresses to market-ready technology, additional capability becomes available to reduce the percentage of pipelines that cannot be inspected. One technology that falls into this category is high-power “brush-drive” pipeline crawlers. Pipeline crawlers that use reciprocating brushes as their source of grip and motive force have been in development for more than 10 years. Fully developed “tethered” production tools are now at the stage of field trials and early operations with a number of companies. Self powered and with high grip and high pull loads at the disposal of the crawler, they are capable of operating in a pipeline independently of most flow conditions, can operate in complex geometries, can be deployed and recovered from a single access point, and can be integrated with existing intelligent inspection tools and cleaning equipment. In essence, they are an enabling technology that can transport existing and established technology used in conventional pigging into an until-now unpiggable part of the pipeline network. Traversing Pipelines Propulsion. The crawler tool uses a free-running “brush drive” attached over the body of a reciprocating device, attached to a fixed-brush section. The brushes are designed to be larger than the pipe ID so they will bend to conform to the pipe bore. In this condition, the brushes possess a gripping force in one direction that is much larger than its sliding force in the opposite direction. When the brush drive is moved backward by the reciprocator (drive motor), the grip force of the brush moves the crawler tool along the pipe. When the brush drive is moved backward by the reciprocator, the fixed-brush grip force exceeds that of the drive brush and the drive brush moves forward. By repeating this process, a brush-driven crawler can travel forward.