-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to debug spider? #40
Comments
For now the only way to get feedbacks while scrapping is to run it with the terminal open (for instance having your local instance run from the terminal and checking the outputs, or checking the log files)... Could you share your scraper config (screenshot) to get an idea how you had your first try ? |
Hi @thibault, good to see you here :-) |
@thibault I tried with that :
I got nothing weird in my log, no error message, but the page is not loaded...
Very weird indeed Meanwhile you can start to try with this website to check if it's the code or the website creating trouble : |
... I added the
and no problem... So it's Scrapy or the website |
@thibault So you could either comment this same line on your instance, or change the |
@JulienParis Wow, it seems I gave you work for the entire afternoon :) Thank you for taking the time to help. I will try your solution, and will get back to you with the results. @DavidBruant Hi ! :) |
Hi @JulienParis,
I'm testing my own instance of OpenScraper.
So far, despite reading the documention, I've been unable to get any real data out of OpenScraper.
I've defined a simple data model (one field), added a simple contributor, but when I "Crawl" the spider, the dataset stays empty.
Now, I'm not too sure where to go from here. I've tested and re-tested my xpaths expressions, and although I might be wrong, it seems to me everything is ok here. How do I get feedback about the scraping results? How do I know what happened during the scrolling and what went wrong exactly?
The text was updated successfully, but these errors were encountered: