An automated Bot that enters the User onto Giveaways on Goodreads using Scrapy web-crawler.
- The user needs to have a Goodreads Account.
- The user needs to have atleast one shipping address saved beforehand.
To run the bot, Scrapy needs to be installed on the system.
To install Scrapy using conda, run:
$ conda install -c conda-forge scrapy
Alternatively, installation through PyPI:
$ pip install Scrapy
Run the crawler
$ scrapy crawl giveaway -a username="***email_id***" -a password="***password***"
This will start the bot with User logging in using the Email ID and password provided as arguments. Once logged in, the spider will start, traversing through the Giveaway pages : Ending soon , Most requested, Popular authors and the Latest giveaways
-
Asychronous calls to pages, meaning faster crawling
-
Verbose Output during execution of Spider
-
Option to Ignore giveaways based on selective keywords as specified by user
- Blacklist according to the title
# blacklisted_titles.txt # provide the list of blacklisted words that should be checked in the titles # provide each word line-wise # Ex: word1 word2 word3
- Blacklist according to the description
# blacklisted_words.txt # provide the list of blacklisted words that should be checked in the decription # provide each word line-wise # Ex: word11 word12 word13
-
Store the Entered Giveaways list in file : #EnteredGiveaways.txt
-
Store the Rejected Giveaways list in file : #RejectedGiveaways.txt
-
When the Giveaway is being entered, the "Also add this book to my to-read shelf." checkbox is un-ticked.
-
# Obey robots.txt rules ROBOTSTXT_OBEY = True # Enable and configure the AutoThrottle extension AUTOTHROTTLE_ENABLED = True #Enable Cookies that are received and keeps track of cookies sent by servers, #and sends them back on subsequent requests #COOKIES_ENABLED = True