查看文章

The large size and the dynamic nature of the Web highlight the need for continuous support and updating of Web based information retrieval systems. Crawlers facilitate the process by following the hyperlinks in Web pages to automatically download a partial snapshot of the Web. While some systems rely on crawlers that exhaustively crawl the Web, others incorporate focus within their crawlers to harvest application or topic specific collections. This project studied web crawling and scraping at many different levels. It will aggregate information from multiple sources into one central location. It Specifics a program for downloading web pages. Given an initial set of seed URLs, it recursively downloads every page that is linked from pages in the set, that have content satisfies specific criterion. Social media, web applications, and mobile applications have been employed together in the proposed system to manage the …