Duplicate Web Pages Research Articles

A Study the review of Duplicate Data in Cloud Computing

In the realm of cloud computing, the management of data redundancy, often referred to as duplicate data, poses a significant challenge due to its implications on storage efficiency, data integrity, and overall system performance. This study provides a comprehensive review of the current strategies employed to detect, manage, and eliminate duplicate data in cloud environments. Duplicate and nearly duplicate web pages cause major issues for web search engines. Duplicate document identification, or the process of finding document pairs that represent the same entity, is a basic component of data cleansing. In order to address the structural heterogeneity issue, data must first undergo a data preparation step that involves parsing, data transformation, and data standardization. The findings aim to aid cloud service providers and users in understanding the importance of effective data deduplication strategies to optimize storage resources and enhance cloud computing performance.

Application of Artificial Intelligence to Patient-Targeted Health Information on Kidney Stone Disease

The American Medical Association (AMA) recommends health information to be written at a 6th grade level reading level. Our aim was to determine whether Artificial Intelligence (AI) can outperform the existing health information on kidney stone prevention and treatment. The top 50 search results for "Kidney Stone Prevention" and "Kidney Stone Treatment" on Google, Bing, and Yahoo were selected. Duplicate webpages, advertisements, pages intended for health professionals such as science articles, links to videos, paid subscription pages, and links non-related to kidney stone prevention and/or treatment were excluded. Included pages were categorized into academic, hospital-affiliated, commercial, non-profit foundations, and other. Quality and readability of webpages were evaluated using validated tools, and the reading level was descriptively compared with ChatGPT generated health information on kidney stone prevention and treatment. 50 webpages on kidney stone prevention and 49 on stone treatment were included in this study. The reading level was determined to equate to that of a 10th to 12th grade student. Quality was measured as "fair" with no pages scoring "excellent" and only 20% receiving a "good" quality. There was no significant difference between pages from academic, hospital-affiliated, commercial, and non-profit foundation publications. The text generated by ChatGPT was considerably easier to understand with readability levels measured as low as 5th grade. The language used in existing information on kidney stone disease is of subpar quality and too complex to understand. Machine learning tools could in generating information that is comprehensible by the public.

URL’S Phishing Detection Based on Machine Learning Approach

Phishing detection is a challenging problem, and many different solutions are proposed in a market as a blacklist, rule-based detection, anomaly-based detection etc. Phishing Websites are duplicate webpages created to mimic real websites in-order to deceive people to get their personal information. Because of the adaptability of their tactics with little cost detecting and identifying phishing websites is really a obscure and dynamic problem.

On the Commonly-Used Incorrect Visual Representation of Accuracy and Precision

We performed a Google image search using the search phrase “accuracy and precision” and, after removing unresponsive and duplicate web pages from the search results, found that 78 of the top 100 results use the bullseye chart to visually explain this concept. Unfortunately, we also found that 52 of those 78 results, i.e., an incredible two thirds, use a visual similar to what is shown in Fig. 1(i) or Fig. 1(ii), both of which are at best highly misleading and factually incorrect according to relevant standards and guidelines. In this figure, the black dots represent the values obtained by replicated measurements of the same measurand (the quantity intended to be measured), and the visual aims to present how close the measured values are to each other and to an agreed-upon reference value, represented by the red bullseye. In the search results, visuals similar to Fig. 1(i) and Fig. 1(ii) appeared 25 and 27 times, respectively. In the top 20 search results, which is where the great majority of users look for answers, the earliest correct visual appeared only at position 11, followed by 13, 15, and 19 in the rankings. The great majority of the 52 incorrect visuals were in non-peer-reviewed documents, reinforcing the notion that one should not believe everything one sees on the internet, although shockingly, a few were in papers published in peer-reviewed scientific venues, including one in Elsevier's Journal of Clinical Epidemiology and another one in the SCMR's Journal of Cardiovascular Magnetic Resonance, both well-cited journals. The actual numbers are probably higher, as many scientific articles are behind pay-walls and inaccessible to Google search.

Pelvic vein embolization: an assessment of the readability and quality of online information for patients

IntroductionPelvic congestion syndrome is a controversial topic. Pelvic vein embolization is a minimally invasive treatment for pelvic congestion syndrome. We aimed to assess the quality of information available on the Internet and determine how accessible information provided by the main IR societies was to patients.Materials and methodsThe most commonly used term relating to pelvic vein embolization was searched across the five most-used English language search engines, with the first 25 web pages returned by each engine included for analysis. Duplicate web pages, nontext content and web pages behind paywalls were excluded. Web pages were analyzed for quality and readability using validated tools: DISCERN score, JAMA Benchmark Criteria, HONcode Certification, Flesch Reading Ease Score, Flesch–Kincaid Grade Level, and Gunning–Fog Index.ResultsThe most common applicable term was “Pelvic Vein Embolization”. Mean DISCERN quality of information provided by websites is “fair”. Flesh–Kincaid readability tests and Gunning–Fog Index demonstrated an average “college level” of reading ease. HON code certification was demonstrated in less than one third of web pages. Professional societies and scientific journals demonstrated the highest average JAMA and DISCERN scores, while for-profit organizations and healthcare providers demonstrated the lowest. Only information from 1 of 3 interventional societies was included in the first 25 search engine pages.ConclusionThe quality of information available online to patients is “fair” and outside of scientific journals the majority of web pages do not meet the JAMA benchmark criteria. These findings call for the production of high-quality and comprehensible content regarding interventional radiology, where physicians can reliably direct their patients for information.

SETJoin: a novel top-k similarity join algorithm

As an important operation in data cleaning, near duplicate Web pages detection and data mining, similarity joins have received much attention recently. Existing similarity joins fall into two broad categories—the similarity-threshold-based similarity join and top-k similarity join (TopkJoin). Compared with the traditional one, TopkJoin is more suitable for cases where the similarity threshold is unknown before hand. In this paper, we focus on the performance optimization problem of TopkJoin. Particularly, we observed that the state-of-the-art TopkJoin algorithm has three serious performance issues, i.e., the inappropriate application of hash table, inefficient use of suffix filtering and unnecessary evaluation of excessive unqualified candidates. To resolve these problems, we proposed a novel algorithm, SETJoin, by combining the existing event-driven framework with three simple yet efficient optimization techniques, viz., (1) reducing the cost in hashing by rearranging the orders of the candidate filtering and hash table lookup operations; (2) maximizing the pruning capability of suffix filtering by judiciously choosing the (near) optimal recursion depth; and (3) terminating join operations earlier by setting a much tighter stop condition for iteration. The experimental results show that SETJoin achieves up to 1.26x–3.49x speedup over the state-of-the-art algorithm on several real datasets.

Varicocele Embolization: An Assessment of the Quality and Readability of Online Patient Information

Varicocele embolization is a growing treatment modality owing to the safety, efficacy, and quick return to work following the procedure. The internet is the most dominant source of information for many. We aimed to assess the quality of information accessible by patients considering treatment. A list of applicable, commonly used searchable terms was generated. Each term was assessed across the five most-used English language search engines to determine the two most commonly used terms. These two terms were then investigated across each search engine, with the first 25 web pages returned by each engine included for analysis. Duplicate web pages, nontext content such as video or audio, and web pages behind paywalls were excluded. Web pages were analyzed for quality and readability using validated tools including DISCERN score, JAMA Benchmark Criteria, HONcode Certification, Flesch Reading Ease Score, Flesch-Kincaid Grade Level, and Gunning-Fog Index. Secondary features including age, rank, author, and publisher were recorded. The most common applicable terms were "Testicular embolization" (378,300 results) and "Varicocele embolization" (375,800 results). Mean DISCERN quality of information provided by websites is "fair"; Adherence to JAMA Benchmark Criteria was 13.5%. Flesh-Kincaid readability tests demonstrated an average "9th grade" reading level. Scientific journals showed the highest quality scores, but were least up to date with an average web page age of 11.2 years. Web pages produced by "for-profit" organizations were the second most current (average age 2.7 years), but displayed the lowest quality of information scores. While quality of online information available to patients is "fair," adherence to JAMA benchmark criteria is poor. "For-profit" organization websites are far more numerous and significantly more up-to-date, yet showed significantly lower quality of information scores. Scientific journals were unsurprisingly of higher quality, yet more challenging for the general public to read. These findings call for the production of high-quality and comprehensible content regarding interventional radiology, where physicians can reliably direct their patients for information.

A NOVEL TWO-PHASE PAGE FEATURE AND KTH KEYPHRASE FINGERPRINT BASED DUPLICATE DETECTION TECHNIQUE

The World Wide Web is a huge repository of network-accessible information including text, image, audio, video and metadata. With rapid increase in information resources available via WWW and users of the Internet, it is becoming difficult to manage and access the desired information on the web. Therefore, majority of users use information retrieval tools like search engines to find the desired information from the WWW. Web search engines work by storing information about many web pages, which they retrieve from the WWW itself.. Many of the pages stored in search engine repository are duplicates and near duplicates of other pages. These duplicate and near duplicate web pages require more space for storage, which increase the cost of serving results and also frustrates the users. To help search engines provide quality and redundant free, distinct results duplicate and near duplicate detection algorithms are used. The proposed duplicate detection approach detects near duplicate web pages efficiently and quickly thereby improving search effectiveness and storage efficiency of search engine.

Natural Language Semantic Construction Based on Cloud Database

Natural language semantic construction improves natural language comprehension ability and analytical skills of the machine. It is the basis for realizing the information exchange in the intelligent cloud-computing environment. This paper proposes a natural language semantic construction method based on cloud database, mainly including two parts: natural language cloud database construction and natural language semantic construction. Natural Language cloud database is established on the CloudStack cloud-computing environment, which is composed by corpus, thesaurus, word vector library and ontology knowledge base. In this section, we concentrate on the pretreatment of corpus and the presentation of background knowledge ontology, and then put forward a TF-IDF and word vector distance based algorithm for duplicated webpages (TWDW). It raises the recognition efficiency of repeated web pages. The part of natural language semantic construction mainly introduces the dynamic process of semantic construction and proposes a mapping algorithm based on semantic similarity (MBSS), which is a bridge between Predicate-Argument (PA) structure and background knowledge ontology. Experiments show that compared with the relevant algorithms, the precision and recall of both algorithms we propose have been significantly improved. The work in this paper improves the understanding of natural language semantics, and provides effective data support for the natural language interaction function of the cloud service.

Detection and elimination of similar Web pages based on text structure and string of feature code

In order to reduce the interference of the duplicated Web pages,and improve the efficiency of detection and elimination of similar Web pages,a new kind of large-scale Web page detection algorithm was proposed.Firstly,adopting the Web label values,the algorithm created the text structure trees to realize the fingerprint similarity calculation layer by layer.Secondly,the head and tail words of a certain sentence,in which high frequency punctuations occur,were extracted out as the feature code.Lastly,the fingerprint similarity of Web page features was discriminated with Bloom filter algorithm.The experimental results show that the algorithm can improve the recall rate up to more than 90%,and reduce the time complexity to O(n).

Near Duplicate Web Page Detection using NDupDet Algorithm

is a system of interlinked hypertext documents accessed via Internet. Internet is a global system of interconnected computer networks that serve billions of users worldwide. The huge amount of documents on the web is challenging for web search engines. Web contains multiple copies of the same content or same web page. Many of these pages on the Web are duplicates and near duplicates of other pages. Web search engines face substantial problems due to such duplicate and near duplicate web pages. These pages enlarge the space required to store the index, increase the cost of serving results and so frustrates the users. To assist search engines to provide search results free of redundancy to users and to provide distinct useful results on the first page, duplicate and near duplicate detection is required. The proposed approach will detect near duplicate web pages to increase search effectiveness and storage efficiency of search engine.

The Chinese Duplicate Web Pages Detection Algorithm based on Edit Distance

On one hand, redundant pages could increase searching burden of the search engine. On the other hand, they would lower the user’s experience. So it is necessary to deal with the pages. To achieve near-replicas detection, most of the algorithms depend on web page content extraction currently. But the cost of content extraction is large and it is difficult. What’s more, it becomes much harder to extract web content properly. This paper addresses these issues through the following ways: it gets the definition of the largest number of common character by taking antisense concept of edit distance; it suggests that the feature string of web page built by a previous Chinese character of period in simple processing text; and it utilizes the largest number of common character to calculate the overlap factor between the feature strings of web page. As a consequence, this paper hopes to achieve near-replicas detection in high noise environment, avoiding extracting the content of web page. The algorithm is proven efficient in our experiment testing: the recall rate of web pages reaches 96.7%, and the precision rate reaches 97.8%.

Near-Duplicate Web Page Detection: An Efficient Approach Using Clustering, Sentence Feature and Fingerprinting

Abstract Duplicate and near-duplicate web pages are the chief concerns for web search engines. In reality, they incur enormous space to store the indexes, ultimately slowing down and increasing the cost of serving results. A variety of techniques have been developed to identify pairs of web pages that are “similar” to each other. The problem of finding near-duplicate web pages has been a subject of research in the database and web-search communities for some years. In order to identify the near duplicate web pages, we make use of sentence level features along with fingerprinting method. When a large number of web documents are in consideration for the detection of web pages, then at first, we use K-mode clustering and subsequently sentence feature and fingerprint comparison is used. Using these steps, we exactly identify the near duplicate web pages in an efficient manner. The experimentation is carried out on the web page collections and the results ensured the efficiency of the proposed approach in dete...

Performance and Comparative Analysis of the Two Contrary Approaches for Detecting Near Duplicate Web Documents in Web Crawling

Recent years have witnessed the drastic development of World Wide Web (WWW). Information is being accessible at the finger tip anytime anywhere through the massive web repository. The performance and reliability of web engines thus face huge problems due to the presence of enormous amount of web data. The voluminous amount of web documents has resulted in problems for search engines leading to the fact that the search results are of less relevance to the user. In addition to this, the presence of duplicate and nearduplicate web documents has created an additional overhead for the search engines critically affecting their performance. The demand for integrating data from heterogeneous sources leads to the problem of near-duplicate web pages. The detection of near duplicate documents within a collection has recently become an area of great interest. In this research, we have presented an efficient approach for the detection of near duplicate web pages in web crawling which uses keywords and the distance measure. Besides that, G.S. Manku et al.’s fingerprint based approach proposed in 2007 was considered as one of the “state-of-the-art algorithms for finding near-duplicate web pages. Then we have implemented both the approaches and conducted an extensive comparative study between our similarity score based approach and G.S. Manku et al.’s fingerprint based approach. We have analyzed our results in terms of time complexity, space complexity, Memory usage and the confusion matrix parameters. After taking into account the above mentioned performance factors for the two approaches, the comparison study clearly portrays our approach the better (less complex) of the two based on the factors considered. General Terms Data Mining, Web Mining

Performance and Comparative Analysis of the Two Contrary Approaches for Detecting Near Duplicate Web Documents in Web Crawling

Recent years have witnessed the drastic development of World Wide Web (WWW). Information is being accessible at the finger tip anytime anywhere through the massive web repository. The performance and reliability of web engines thus face huge problems due to the presence of enormous amount of web data. The voluminous amount of web documents has resulted in problems for search engines leading to the fact that the search results are of less relevance to the user. In addition to this, the presence of duplicate and near-duplicate web documents has created an additional overhead for the search engines critically affecting their performance. The demand for integrating data from heterogeneous sources leads to the problem of near-duplicate web pages. The detection of near duplicate documents within a collection has recently become an area of great interest. In this research, we have presented an efficient approach for the detection of near duplicate web pages in web crawling which uses keywords and the distance measure. Besides that, G.S. Manku et al.’s fingerprint based approach proposed in 2007 was considered as one of the “state-of-the-art algorithms for finding near-duplicate web pages. Then we have implemented both the approaches and conducted an extensive comparative study between our similarity score based approach and G.S. Manku et al.’s fingerprint based approach. We have analyzed our results in terms of time complexity, space complexity, Memory usage and the confusion matrix parameters. After taking into account the above mentioned performance factors for the two approaches, the comparison study clearly portrays our approach the better (less complex) of the two based on the factors considered. DOI: http://dx.doi.org/10.11591/ijece.v2i6.1746 Full Text: PDF

An Efficient Approach for Finding Near Duplicate Web pages using Minimum Weight Overlapping Method

The existence of billions of web data has severely affected the performance and reliability of web search. The presence of near duplicate web pages plays an important role in this performance degradation while integrating data from heterogeneous sources. Web mining faces huge problems due to the existence of such documents. These pages increase the index storage space and thereby increase the serving cost. By introducing efficient methods to detect and remove such documents from the Web not only decreases the computation time but also increases the relevancy of search results. We aim a novel idea for finding near duplicate web pages which can be incorporated in the field of plagiarism detection, spam detection and focused web crawling scenarios. Here we propose an efficient method for finding near duplicates of an input web page, from a huge repository. A TDW matrix based algorithm is proposed with three phases, rendering, filtering and verification, which receives an input web page and a threshold in its first phase, prefix filtering and positional filtering to reduce the size of record set in the second phase and returns an optimal set of near duplicate web pages in the verification phase by using Minimum Weight Overlapping (MWO) method. The experimental results show that our algorithm outperforms in terms of two benchmark measures, precision and recall, and a reduction in the size of competing record set. DOI: http://dx.doi.org/10.11591/ijece.v1i2.77

Near Duplicated Web Pages Detection Based on Concept and Semantic Network

在搜索引擎的检索结果页面中,用户经常会得到内容近似的网页.为了提高检索整体性能和用户满意度,提出了一种基于概念和语义网络的近似网页检测算法DWDCS(near-duplicate webpages detection based on concept and semantic network).改进了经典基于小世界理论提取文档关键词的算法.首先对文档概念进行抽取和归并,不但解决了“表达差异”问题,而且有效降低了语义网络的复杂度;从网络结构的几何特征对其进行分析,同时利用网页的语法和结构信息构建特征向量进行文档相似度的计算,由于无须使用语料库,使得算法天生具有领域无关的优点.实验结果表明,与经典的网页去重算法(I-Match)和单纯依赖词汇共现小世界模型的算法相比,DWDCS 具有很好的抵抗噪声的能力,在大规模实验中获得了准确率>90%和召回率>85％的良好测试结果.良好的时空间复杂度及算法性能不依赖于语料库的优点,使其在大规模网页去重实际应用中获得了良好的效果.;Reprinting websites and blogs produces a great deal redundant WebPages. To improve search efficiency and user satisfaction, the near-Duplicate WebPages Detection based on Concept and Semantic network (DWDCS) is proposed. In the course of developing a near-duplicate detection system for a multi-billion pages repository, this paper makes two research contributions. First, the key concept is extracted, instead of the keyphrase, to build Small Word Network (SWN). This not only reduces the complexity of the semantic network, but also resolves the “expression difference” problem. Second, this paper considers both syntactic and semantic information to present and compute the documents’ similarities. In a large-scale test, experimental results demonstrate that this approach outperforms that of both I-Match and keyphrase extraction algorithms based on SWN. Many advantages such as linear time and space complexity, without using a corpus, make the algorithm valuable in actual practice.

Research of a Novel P2P Search Algorithm Based on Small-World Phenomena

This paper proposes a novel breadth-first search algorithm and deals with the problem of duplicate web pages removing and page ranking by the principle of Small World phenomena. The features of algorithm in this paper are as follows: 1) it proposes the Unit Tree-based Breadth-First Search Algorithm and performs a qualitative analysis and simulated calculation of the performance, with the results demonstrating that the algorithm in this paper reduces the number of neighboring nodes to which messages shall be forwarded and redundant messages as compared with traditional Breath-First Search algorithm and Random Breath-First Search algorithm; 2) it proposes a solution to the problem of duplicate web pages removing and page ranking by Small World phenomena, with the results demonstrating that the method in this paper can effectively bring processing workload, waiting time and the pointless network bandwidth consumption of source node searching into balance.

Correlation Based Method to Detect and Remove Redundant Web Document

The enrichment of internet has resulted in the flooding of abundant information on WWW with more replicas. As the duplicated web pages increase the indexing space and time complexity, finding and removing these pages becomes significant for search engines and other likely system which will improve on accuracy of search results as well as search speed. Web content mining plays a vital role in resolving these aspects. Existing algorithm for web content mining focus attention on applying weightage to structured documents whereas in this research work, a mathematical approach based on linear correlation is developed to detect and remove the duplicates present in both structured and unstructured web document. In the proposed work, linear correlation between two web documents is found out. If the correlated value is 1 then the documents are said to be exactly redundant and it should be eliminated otherwise not redundant.

New algorithm based on repeat sequence deletion

Aiming at the current de-duplication algorithms,two repeated sequences(RS)extracting algorithms were compared and analyzed.Since STC has favorable performance in considering time cost and the inverted index method is superior in terms of spatial complexity,STC was used to improve RS algorithm.Experiment results show that this method can find similar Web pages efficiently.This algorithm can reach a high precision in mono-language deletion of duplicated Web pages,and this algorithm can also reach a maximum precision when it is applied to deletion of duplicated web pages.