intern-supply-scraper

Update (Jan. 11, 2017): It looks like Intern Supply isn't being updated as actively as I'd hoped it'd be when developing this. For this reason, I'm going to decrease the frequency of dumps to once/week (instead of daily).

A scraper for Intern Supply.

• Data
• Installation
• Usage
• Contribute

Data

New data is scraped and committed daily. Find the set of scraped data here.

Installation

• Requirements:

  • Go
  • Git

• Clone repository

  $ git clone https://github.com/kshvmdn/intern-supply-scraper.git
  $ cd intern-supply-scraper
  

• Install dependencies

  $ go get
  

Usage

• The barebone scraper will generate a CSV file on every run and commit that once every 24 hours.

• Run the scraper:

  $ go run scraper.go [OPTIONS]

• If you choose, you can build first, then run:

$ go build scraper.go
$ ./scraper [OPTIONS]

• Command line flags:

--no-commit     Omit the git pull/add/commit/push step.
--verbose       Get detailed logging output.
--diff          Run git diff on the 2 latest dump files. Requires there to be at least 2 files present.

Contribute

The project is completely open source. Feel free to open a issue or submit a pull request! 😄