How link scores affect the scheduling

[chris-zen]

Hi, I didn't find any documentation on how the link scores affect/influence their scheduling. It would be nice to understand the relation between:

• spider defined score for the links
• link analysis scorer (PR/HITS)
• scheduler prioritization (BFS, Freq)

Thanks

[plafl]

Until the documentation is updated: in the case of PageRank the algorithm used is Personalized PageRank as originally described here. This means that the higher the spider defined score the higher the probability of a random jump to the page.

The job of the scheduler is to decide what to do with the final scores (spider score + PageRank/HITS). In case of the BFS it simply picks the highest scored web page that is still uncrawled. In case of the FreqScheduler this information is ignored and it simply tries to (re)crawl the pages with the desired frequency: if a page has frequency 8 and the other 2 then the first one is crawled 4 times more often.

[chris-zen]

Thanks @plafl, very informative.