feat: add kingfisher collect log warnings with scrapyloganalyzer #385
Conversation
|
I'm not sure why gunicorn[setproctitle] and yapw[perf] are being removed when running |
|
It's okay - they should be removed. |
| logger.warning("%s: %s", self, {scrapy_log.logparser["log_categories"][key]["details"]}) | ||
| if scrapy_log.error_rate: | ||
| logger.warning("%s: crawl error rate was %s", self, {scrapy_log.error_rate}) |
There was a problem hiding this comment.
Log warnings don't generate emails and aren't saved (except temporarily in server log files).
If you want to generate emails, you need to use the error level.
If you want to save the warnings, you need to update self.job.context.
There was a problem hiding this comment.
Oh, nevermind, the cron job sets the log level to WARNING, so we will get emails.
Can you check whether error_rate can have a division by zero error when no items were scraped? Ideally with a test in scrapy-log-analyzer.
Also, when the error_rate is non-zero, should the message be more detailed, e.g. print the error messages and/or error counts? https://kingfisher-collect.readthedocs.io/en/latest/logs.html#check-the-crawl-statistics
I think we should also check that scrapy_log.is_finished() is True. (This is different from the Scrapyd job being finished.) The log message can include logparser["finish_reason"]. https://kingfisher-collect.readthedocs.io/en/latest/logs.html#check-the-reason-for-closing-the-spider
In open-contracting/scrapy-log-analyzer#11 we discussed storing item_dropped_count and item_scraped_count in the job context (easier to analyze multiple jobs without having to run logreport manually). If there are already other warnings, these can be logged as well.
I think that's everything from open-contracting/scrapy-log-analyzer#11, but please have a look over again in case I missed an earlier idea.
There was a problem hiding this comment.
Hmm ok, I wasn't sure how much of what we discussed there we wanted to add to the Registry vs to a Kingfisher Collect command. But I guess is easier to have the full info in the Data Registry too
There was a problem hiding this comment.
Can you check whether error_rate can have a division by zero error when no items were scraped? Ideally with a test in scrapy-log-analyzer.
Do you mean with outputs like these? If so, the error rate is 1 (I will push the test if that is what you meant)
There was a problem hiding this comment.
Aha, I guess there is always going to be at least one error.
Yeah, my thinking is it will be faster, operationally, to have the same info directly available in the Registry admin. Once the Collect command is implemented, we can see how to reduce any code duplication at that time.
ref #29