Abstract
Efficient access to timely information is critical in today's digital era. Web crawlers, automated programs that navigate the Internet, play an important role in collecting data from websites such as Kumparan, a leading news site in Indonesia. This research shows the effectiveness of the Breadth-First Search (BFS) and Depth-First Search (DFS) algorithms in indexing Kumparan content. The results of the research show that BFS consistently indexes more files comprehensively but with longer execution times compared to DFS, which provides faster initial results but with fewer files. For example, at depth 4 BFS indexed 949 files in 886.94 seconds, while DFS indexed 470 files in 233.02 seconds. These findings highlight the balance between precision and speed when selecting a crawling algorithm tailored to the needs of a particular website. This research provides insights into optimizing web crawler technology for complex websites such as Coil and suggests avenues for further research to improve permission efficiency and adaptability across a variety of crawling scenarios.
Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
