update environment and notebook 2 with dask-searchcv

[rdvelazquez]

Here's the very simple PR to add the dask-searchCV to the environment and dask-searchCV's implementation of GridSearchCV to notebook 2.

A few notes:

1. I evaluated the speed gains of dask-searchCV in Upload dask-searchCV notebook #101 and dask-searchCV seemed to make GridSearchCV about 3 times faster. The wall times for GridSearchCV in this PR are similar to those from @dhimmel 's notebook 2 times with the SK-learn GridSearchCV not because dask-searchCV doesn't speed things up but because my computer is slow. :(
2. It's probably worth having some one else try to update their conda environment and run notebook 2 to confirm that everything works.
3. I didn't rename GridSearchCV... I debated renaming it to include "dask" or "dask-searchCV" (by using from dask_searchcv import GridSearchCV as dask-GridSearchCV) as this would make it easier for someone to see that it wasn't the SK-learn implementation but I thought there was potentials for confusion either way so I would just keep it the same.
4. Just as an FYI - I tried using dask_searchcv in the marginal gain notebook and got the following error at line 30:

# Train with covariates data
all_best_estimator_aurocs_covariates, all_grid_aurocs_covariates = train_and_evaluate(covariates, pipeline)


AttributeError: 'GridSearchCV' object has no attribute 'grid_scores_'

I'm not sure what caused this. I searched the dask-searchcv github page for 'grid_scores_' and didn't see anything. If this becomes an issue in the future we can look into it further or revert that specific case to SK-learn's GridSearchCV.

Closes #94

[dhimmel]

I agree that from dask_searchcv import GridSearchCV is the most pythonic implementation. This is supposed to be swapped in for GridSearchCV so it makes sense not to change the name.

Regarding the 'GridSearchCV' object has no attribute 'grid_scores_', that's expected. See http://scikit-learn.org/stable/whats_new.html#model-selection-enhancements-and-api-changes:

The new cv_results_ attribute (of model_selection.GridSearchCV and model_selection.RandomizedSearchCV) introduced in lieu of the grid_scores_ attribute is a dict of 1D arrays with elements in each array corresponding to the parameter settings (i.e. search candidates).

[dhimmel]

Do you have any indication how dask-searchcv changed the runtime on your machine?

Here are the results with dask-searchcv:

Fitting CV for model: full
	runtime: 0:03:09.570861
Fitting CV for model: expressions
	runtime: 0:03:14.775702
Fitting CV for model: covariates
	runtime: 0:00:27.543525

Also github not showing diff for 2.mutation-classifier.ipynb. Assuming an image changed causing large diff. Any idea what changed from a local git diff?

[dhimmel]

There should be a newline at the end of file, but no spaces on the newline

[dhimmel]

Let's also specify a dask channel and version

[rdvelazquez]

I think a version number for dask-searchcv would be good. Are you also thinking we would explicitly import dask? So it would be:

conda-forge::dask=0.14.3
conda-forge::dask-searchcv=0.0.2

[dhimmel]

Explicitly add dask-searchcv and dask to environment.yml both with versions. As far as importing dask in the notebook, only if it's necessary. Also consider using anaconda dask unless that doesn't work. Currently, we're using the anaconda channel over conda-forge in terms of priority.

[rdvelazquez]

I added version numbers to both dask and dask-searchcv and used anaconda for dask. Everything works fine.

[rdvelazquez]

Do you have any indication how dask-searchcv changed the runtime on your machine?

dask-searchcv was about twice as fast. Average of three runs without dask-searchcv = 14.5 minutes; average of three runs with dask-searchcv = 6.5 minutes.

Any idea what changed from a local git diff?

Is this what you're looking for?

+++ b/2.mutation-classifier.ipynb
@@ -23,7 +23,8 @@
     "from sklearn.decomposition import PCA\n",
     "from sklearn.linear_model import SGDClassifier\n",
     "from sklearn.metrics import roc_auc_score, roc_curve\n",
-    "from sklearn.model_selection import train_test_split, GridSearchCV, Strati
fiedKFold\n",
+    "from sklearn.model_selection import train_test_split, StratifiedKFold\n",
+    "from dask_searchcv import GridSearchCV\n",
     "from sklearn.pipeline import Pipeline, FeatureUnion\n",
     "from sklearn.preprocessing import StandardScaler, FunctionTransformer\n",
     "from vega import Vega\n",
@@ -85,14 +86,15 @@
   {
    "cell_type": "code",
    "execution_count": 4,
-   "metadata": {},
+   "metadata": {
+    "collapsed": false
+   },
    "outputs": [

[dhimmel]

Is this what you're looking for?

Yeah. Just make sure all changes in there make sense.

[dhimmel]

dask-searchcv was about twice as fast. Average of three runs without dask-searchcv = 14.5 minutes; average of three runs with dask-searchcv = 6.5 minutes.

Nice, hopefully we also reduced memory footprint. Was this with only 1 core? In production, we could parallelize.

[rdvelazquez]

Just make sure all changes in there make sense.

Both changes make sense to me: The first change was importing GridSearchCV from dask_searchcv. The second change was not something I edited but rather the output of how long it took to load the data. My notebook only outputted wall time whereas yours outputted CPU times as well. This is likely just a result of running my notebook on Windows (Related StackOverflow Question)

[rdvelazquez]

Was this with only 1 core?

Yep, only 1 core. It will be cool to see how fast this runs in production!
The slowest part of the notebook for me is now loading the data. It may soon be time to revisit pickle (#85).