Adaptive Web Crawlers for Intelligent data extractions | Data Science | Scoop.it

In intelligent adaptive crawling process, the web crawler identifies pages that are updated more frequently by machine-learning. As a radical solution, crawls run more frequently on updated pages than dormant. The crawlers modify automatically to establish optimal frequencies based on site behavior and changes.