add scripts to automatically grow datasets #81

wallnerryan · 2016-10-12T20:27:55Z

these scripts expect data to exit in the app, probably by dataimport initial snapshot.
use fake factory for names and uses vehicle inventory to add more vehicles

Fixes #32
Fixes #49

wallnerryan · 2016-10-12T20:31:04Z

Example Usage

$ ./dataimport/grow-datasets/dealers/Dockerbuild.sh
682a6c45a4b2365ec2d6237ea3f7167037d2cfc8f89e0936444698fe06fe2917

$ ./dataimport/grow-datasets/vehicles/Dockerbuild.sh
fcc7b2304c67417fea962a0af3314d635d8081b469ad2e1a404d721947a9774e

$ docker ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS              PORTS               NAMES
f22cb68e6bcf        clusterhq/add-dealers-loop    "python add_dealers.p"   2 seconds ago       Up 2 seconds                            sleepy_euler
f3fd3823635f        clusterhq/add-vehicles-loop   "python add_vehicles."   17 seconds ago      Up 17 seconds                           agitated_leakey


$:-> docker logs f22cb68e6bcf
Added Dealer: {"phone": "555-555-5555", "name": "Lauren Thompson's, Gregory Smith DDS's & Gwendolyn Mcfarland's Auto Dealership", "addr": "62097 Cox Mission Suite 281 East Brett, NY 57755"}


$:-> docker logs f3fd3823635f
19825
Added Vehicle: {"model": "ACCORD", "year": "2013", "dealership": "a2c1f596-d1fe-4096-849f-b4b4b2f85b96", "make": "HONDA", "vin": "SY2RPPUI8I4X4HIDV"}
17449
Added Vehicle: {"model": "NISSAN FRONTIER 4WD K/C PRO-4X", "year": "2013", "dealership": "2cc315a0-0984-4438-962f-babe40d7bdb4", "make": "NISSAN", "vin": "IK5IU7VLXJZH53RIB"}
24762
Added Vehicle: {"model": "F150 2WD FFV", "year": "2010", "dealership": "3d4cc8e0-10e2-41d3-94ed-f1b5ad91006b", "make": "FORD", "vin": "8VN2LKTN9J3IMOSVB"}
18306
Added Vehicle: {"model": "ES 350", "year": "2011", "dealership": "41a9c461-76e8-4fe0-91d6-480e5fcf55b2", "make": "LEXUS", "vin": "12X1LRJYUXR8Q434H"}
18201
Added Vehicle: {"model": "911 Carrera", "year": "2016", "dealership": "7e2ba903-318c-4072-b832-c5cc6a0e8271", "make": "Porsche", "vin": "ZR15NIG1DB8WAV703"}
6274
Added Vehicle: {"model": "XV CROSSTREK AWD", "year": "2013", "dealership": "13ee2e0b-1f6b-49fd-8c27-05e2084b3a90", "make": "SUBARU", "vin": "9OSRVNSCCIGFSD8SU"}

Vehicles get added every second, and dealers get added every 5 seconds.

REST API URLs are set within the script.

wallnerryan · 2016-10-12T20:31:21Z

cc @pcgeek86

wallnerryan · 2016-10-12T20:54:19Z

We can see how many records returned easily by monitoring the API

https://stedolan.github.io/jq/download/

curl 'http://ec2-54-237-204-239.compute-1.amazonaws.com:32787/dealerships' | jq length

curl 'http://ec2-54-237-204-239.compute-1.amazonaws.com:32787/vehicles' | jq length

wallnerryan · 2016-10-13T13:26:42Z

dataimport/grow-datasets/dealers/add_dealers.py

+
+
+def main():
+    app_url="http://ec2-54-237-204-239.compute-1.amazonaws.com:32787/dealerships"


should be abstracted out to be excepted as input?

wallnerryan · 2016-10-13T13:26:51Z

dataimport/grow-datasets/vehicles/add_vehicles.py

+                print(r.status_code)
+
+def main():
+    dealer_url="http://ec2-54-237-204-239.compute-1.amazonaws.com:32787/dealerships"


should be abstracted out to be excepted as input?

wallnerryan · 2016-10-13T13:26:56Z

dataimport/grow-datasets/vehicles/add_vehicles.py

+
+def main():
+    dealer_url="http://ec2-54-237-204-239.compute-1.amazonaws.com:32787/dealerships"
+    vehicle_url="http://ec2-54-237-204-239.compute-1.amazonaws.com:32787/vehicles"


should be abstracted out to be excepted as input?

…this, eliminate the time sleeps as it was not adding data fast enough

wallnerryan · 2016-10-13T14:48:07Z

rethink_db starts about 50MB with our base snapshot, been running for about 20 min and its at 70M.

Slow road, but eventually we can grow to GB size.

$ du -csh --block-size=1M /chq/1c030aa0-f301-4a42-a19a-8c63c8977f88/9924c5c7-1cb0-45ad-bd03-a2d7b818bca9/*
70  /chq/1c030aa0-f301-4a42-a19a-8c63c8977f88/9924c5c7-1cb0-45ad-bd03-a2d7b818bca9/rethinkdb_data

…s per dealership

wallnerryan · 2016-10-14T14:23:29Z

750,000 Vehicle Records, 760,000 Dealership Records at 580MB.
- https://s3.amazonaws.com/ryanwallner/580MB_750kVehicles_RethinkDB.tar.gz

cc @pcgeek86 you can untar this into /chq/<uuid-of-volset>/<uuid-of-volume>/rethinkdb_data and you can either take a snapshot of it and push to FH if we dont have a snap of it before, or use it to test. I found a lot of issue with our app once we have this much data :)

also, im still working on growing the dataset so its at least 1G

wallnerryan · 2016-10-14T14:33:42Z

It was also hard to get the size from using jq as it download then counted locally. Instead, I added APIs to do that faster.

curl http://ec2-52-90-32-15.compute-1.amazonaws.com:32799/dealershipssize
{"status":"Size: 762311"}

curl http://ec2-52-90-32-15.compute-1.amazonaws.com:32799/vehiclessize
{"status":"Size: 747310"}

…a list of vehicles and dealers without drilling down or hitting timeouts on datasets over 1 million records

add scripts to automatically grow datasets

12bdd54

This was referenced Oct 12, 2016

Grow the dataset size. #49

Closed

create "live" client that can git HTTP endpoints to manipulate data #32

Closed

wallnerryan commented Oct 13, 2016

View reviewed changes

Ryan Wallner added 2 commits October 13, 2016 10:37

make URL as input to script, change dockerfile and script to support …

05a448d

…this, eliminate the time sleeps as it was not adding data fast enough

update comments

6fa368e

Ryan Wallner added 3 commits October 13, 2016 17:31

Merge branch 'master' into grow-dataset

61e3a00

add a refresh loop for vehicles/dealers, remove real dns/port from git

3a382af

use 750k dealerships to test size endpoints and new limits on vehicle…

9ef3ac4

…s per dealership

Ryan Wallner added 13 commits October 14, 2016 10:50

Try and mess with the timeouts

de7c1fc

try and use index for vehicles too since we still have a timeout issue

bfe4721

limit to 30,000 records returned for vehicles

bf56e79

limit to 30,000 records returned for dealers

207cd81

add comments, larger cache size

a5c40ca

skip tests right now, just stage

11a72ab

skip tests right now, just stage to test sample queries

f58f9cf

scale back sample

256c812

try sammple on Table rather than same on OrderBy

7696f71

samples take too long

ecba38f

add a vehicle list page so that even with large datasets we can show …

83f9005

…a list of vehicles and dealers without drilling down or hitting timeouts on datasets over 1 million records

re-enable tests, change width

1a31a8d

remove old comment

e7b9864

refresh every 20k records, isntead of 5k

b5695de

wallnerryan merged commit b7c8c24 into master Oct 14, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add scripts to automatically grow datasets #81

add scripts to automatically grow datasets #81

wallnerryan commented Oct 12, 2016 •

edited

Loading

wallnerryan commented Oct 12, 2016

wallnerryan commented Oct 12, 2016

wallnerryan commented Oct 12, 2016 •

edited

Loading

wallnerryan Oct 13, 2016

wallnerryan Oct 13, 2016

wallnerryan Oct 13, 2016

wallnerryan Oct 13, 2016

wallnerryan Oct 13, 2016

wallnerryan Oct 13, 2016

wallnerryan commented Oct 13, 2016 •

edited

Loading

wallnerryan commented Oct 14, 2016 •

edited

Loading

wallnerryan commented Oct 14, 2016



		def main():
		app_url="http://ec2-54-237-204-239.compute-1.amazonaws.com:32787/dealerships"

add scripts to automatically grow datasets #81

add scripts to automatically grow datasets #81

Conversation

wallnerryan commented Oct 12, 2016 • edited Loading

wallnerryan commented Oct 12, 2016

wallnerryan commented Oct 12, 2016

wallnerryan commented Oct 12, 2016 • edited Loading

wallnerryan Oct 13, 2016

Choose a reason for hiding this comment

wallnerryan Oct 13, 2016

Choose a reason for hiding this comment

wallnerryan Oct 13, 2016

Choose a reason for hiding this comment

wallnerryan Oct 13, 2016

Choose a reason for hiding this comment

wallnerryan Oct 13, 2016

Choose a reason for hiding this comment

wallnerryan Oct 13, 2016

Choose a reason for hiding this comment

wallnerryan commented Oct 13, 2016 • edited Loading

wallnerryan commented Oct 14, 2016 • edited Loading

wallnerryan commented Oct 14, 2016

wallnerryan commented Oct 12, 2016 •

edited

Loading

wallnerryan commented Oct 12, 2016 •

edited

Loading

wallnerryan commented Oct 13, 2016 •

edited

Loading

wallnerryan commented Oct 14, 2016 •

edited

Loading