Adaptive crawl rate

[brendanheywood]

ie if a url works, and has worked consistently, then slowly ramp down how often it is rescraped. The key factors we'd want to consider using are:

• when it was last crawled
• its history of good crawls
• how often it if viewed (uselogs)
• how often it is edited
• if the links going out (and maybe in) have changed

I don't want to use a factor like 'is the course inactive'. We can already do this for free, the robot user should only have access to content it should crawl so this can be managed by role assignments to course categories etc