Docker Image for development

[allanbatista]

Was development two Docker images, one generic with marvin toolbox only and other to dev with spark and user included.

# expose ports 8000 and 8888 to random ports
$ docker run -it -P allanbatista/marvin-dev:0.0.2 /bin/bash -c "su marvin"

# expose ports 8000 and 8888
$ docker run -it -p 8000:8000 -p 8888:8888 allanbatista/marvin-dev:0.0.2 /bin/bash -c "su marvin"

# use toolbox env
$ workon marvin-toolbox-env

# create new engine
$ marvin engine-generate
$ workon NAME-engine-env
$ marvin notebook

[codecov-io]

Codecov Report

Merging #62 into master will increase coverage by 0.7%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master      #62     +/-   ##
=========================================
+ Coverage   55.77%   56.47%   +0.7%     
=========================================
  Files          25       24      -1     
  Lines        2123     2130      +7     
  Branches      217      219      +2     
=========================================
+ Hits         1184     1203     +19     
+ Misses        924      910     -14     
- Partials       15       17      +2

Impacted Files	Coverage Δ
marvin_python_toolbox/common/utils.py	100% <0%> (ø)	⬆️
...thon_toolbox/engine_base/stubs/actions_pb2_grpc.py	40% <0%> (ø)	⬆️
marvin_python_toolbox/common/__init__.py
marvin_python_toolbox/common/data.py	80.7% <0%> (+0.34%)	⬆️
marvin_python_toolbox/management/engine.py	54.59% <0%> (+1.01%)	⬆️
...python_toolbox/engine_base/engine_base_training.py	100% <0%> (+3.33%)	⬆️
...n_python_toolbox/engine_base/engine_base_action.py	70.42% <0%> (+8.45%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c3f4e1e...c6dd193. Read the comment docs.

[oldfly]

Hi @allanbatista, thanks for your PR!

I did some tests, before creating a new engine, if i try to run all unit tests with "marvin test" command, it triggers the IOError about file access permission and I got disconnected from docker immediately.

Am i missing something?

[oldfly]

Error log:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/_pytest/main.py", line 94, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/_pytest/main.py", line 125, in _main
INTERNALERROR> config.hook.pytest_runtestloop(session=session)
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 724, in call
INTERNALERROR> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 338, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 333, in
INTERNALERROR> _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 595, in execute
INTERNALERROR> return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 249, in _wrapped_call
INTERNALERROR> wrap_controller.send(call_outcome)
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/pytest_cov/plugin.py", line 235, in pytest_runtestloop
INTERNALERROR> self.cov_controller.finish()
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/pytest_cov/engine.py", line 115, in summary
INTERNALERROR> total = self.cov.html_report(ignore_errors=True, directory=self.cov_report['html'])
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/coverage/control.py", line 1095, in html_report
INTERNALERROR> return reporter.report(morfs)
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/coverage/html.py", line 139, in report
INTERNALERROR> self.report_files(self.html_file, morfs, self.config.html_dir)
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/coverage/report.py", line 91, in report_files
INTERNALERROR> report_fn(fr, self.coverage._analyze(fr))
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/coverage/html.py", line 286, in html_file
INTERNALERROR> write_html(html_path, html)
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/coverage/html.py", line 69, in write_html
INTERNALERROR> with open(fname, "wb") as fout:
INTERNALERROR> IOError: [Errno 13] Permission denied: u'coverage_report/marvin_python_toolbox_common_data_py.html'

[allanbatista]

Hi @oldfly.

I See. I will resolve soon.

[allanbatista]

@oldfly , fix that problem.

This happen because before complete build dev image, the process run marvin test, but they not create correct permission after.

$ docker run -it -p 8000:8000 -p 8888:8888 allanbatista/marvin-dev:0.0.2 /bin/bash -c "su marvin"
marvin@a2c851c558ea:/opt/marvin$ workon marvin-toolbox-env
(marvin-toolbox-env) marvin@a2c851c558ea:/opt/marvin/toolbox$ marvin test
py.test --cov marvin_python_toolbox --cov-report html --cov-report xml --cov-report term-missing tests
================================================================== test session starts ==================================================================
platform linux2 -- Python 2.7.12, pytest-2.9.2, py-1.5.2, pluggy-0.3.1
rootdir: /opt/marvin/toolbox, inifile: pytest.ini
plugins: testmon-0.9.9, flask-0.10.0, cov-2.5.1
collected 112 items 

tests/test_loader.py .
tests/common/test_config.py ............
tests/common/test_data.py .......
tests/common/test_data_source_provider.py ...
tests/common/test_http_client.py ..............
tests/common/test_profiling.py ...........
tests/common/test_utils.py .......................
tests/engine_base/test_engine_base_action.py .............
tests/engine_base/test_engine_base_data_handler.py ....
tests/engine_base/test_engine_base_prediction.py ....
tests/engine_base/test_engine_base_training.py .....
tests/management/test_engine.py ...
tests/management/test_notebook.py ..
tests/management/test_pkg.py ..........

---------- coverage: platform linux2, python 2.7.12-final-0 ----------
Name                                                            Stmts   Miss Branch BrPart  Cover   Missing
-----------------------------------------------------------------------------------------------------------
marvin_python_toolbox/__init__.py                                   6      0      0      0   100%
marvin_python_toolbox/_compatibility.py                             7      0      0      0   100%
marvin_python_toolbox/_logging.py                                  29      2      0      0    93%   85-86
marvin_python_toolbox/common/__init__.py                            2      0      0      0   100%
marvin_python_toolbox/common/config.py                             65      2     14      0    97%   52-53
marvin_python_toolbox/common/data.py                               56     19     14      2    61%   98-129, 68->70, 97->98
marvin_python_toolbox/common/data_source_provider.py               11      0      2      0   100%
marvin_python_toolbox/common/exceptions.py                          5      0      0      0   100%
marvin_python_toolbox/common/http_client.py                        91      0     12      0   100%
marvin_python_toolbox/common/profiling.py                          94      0     23      0   100%
marvin_python_toolbox/common/utils.py                             135      0     44      0   100%
marvin_python_toolbox/config.py                                    44     33     14      0    19%   37-55, 59-86
marvin_python_toolbox/decorators.py                                 9      9      0      0     0%   18-29
marvin_python_toolbox/engine_base/__init__.py                       5      0      0      0   100%
marvin_python_toolbox/engine_base/engine_base_action.py           142     51     26      3    63%   114-131, 139, 152, 155-160, 163-176, 179-182, 190, 193-198, 201-214, 217-220, 74->79, 136->140, 138->139
marvin_python_toolbox/engine_base/engine_base_data_handler.py      22      0      0      0   100%
marvin_python_toolbox/engine_base/engine_base_prediction.py        22      0      0      0   100%
marvin_python_toolbox/engine_base/engine_base_training.py          30      1      0      0    97%   69
marvin_python_toolbox/engine_base/stubs/__init__.py                 2      0      0      0   100%
marvin_python_toolbox/engine_base/stubs/actions_pb2.py            150     52      2      0    66%   485-495, 509-510, 516-517, 523-524, 529-548, 561-571, 585-586, 592-593, 599-600, 605-624, 638, 642, 646, 680-696, 705-721, 735, 739, 743, 777-793, 802-820
marvin_python_toolbox/engine_base/stubs/actions_pb2_grpc.py        40     24      0      0    40%   17-27, 41-42, 48-49, 55-56, 61-80, 93-103, 117-118, 124-125, 131-132, 137-156
marvin_python_toolbox/loader.py                                     7      0      0      0   100%
marvin_python_toolbox/manage.py                                     1      0      0      0   100%
marvin_python_toolbox/management/__init__.py                       55     38     18      0    23%   64-135
marvin_python_toolbox/management/engine.py                        392    173    112      9    52%   45, 67, 142-143, 207-211, 218-221, 231, 233, 235, 237, 239, 251-272, 294-335, 358-363, 376-451, 458-461, 465-472, 476-505, 509-518, 522-533, 537-543, 547-552, 578, 607-609, 625-628, 648-670, 681-697, 216->218, 230->231, 232->233, 234->235, 236->237, 238->239, 594->599, 596->595, 612->616
marvin_python_toolbox/management/hive.py                          392    315     50      0    17%   38, 44-56, 65-79, 106-146, 150-156, 166-203, 207-223, 226-249, 252-282, 285, 288-291, 297-442, 446, 450, 454, 458, 461, 464-491, 494-497, 500-509, 512-519, 522, 525, 528, 531, 534-537, 540, 546-555, 562-567, 570-571, 574-580, 583-589, 592-598, 601-604, 607-608, 611-612, 615-619, 622-623, 626-627, 630-636, 639-642, 646-674, 677-681
marvin_python_toolbox/management/notebook.py                       18      2      0      0    89%   27, 36
marvin_python_toolbox/management/pkg.py                           297    197     96      0    31%   38, 44, 50, 56-88, 94-135, 148-173, 179-182, 190-235, 241-245, 251-253, 281-294, 298-312, 319-364
marvin_python_toolbox/management/test.py                           80     52     22      0    27%   33, 43-69, 76-90, 101-131, 137-141
-----------------------------------------------------------------------------------------------------------
TOTAL                                                            2209    970    449     14    53%
Coverage HTML written to dir coverage_report
Coverage XML written to file coverage.xml


============================================================== 112 passed in 14.87 seconds ============================================================

Please, do not forget to update image before test again.

docker pull allanbatista/marvin-dev:0.0.2

[oldfly]

Hi @allanbatista, Permission fix checked, thanks!

And after marvin test, the docker container is halted automatically, please check this behavior too!

[takabayashi]

Hi @allanbatista lets finish and merge this code? To do that you must fix the last bug reported by @oldfly ...