step inherits from Task rather than ABC

[donovanr]

Resolves #21
Resolves #28
Resolves #33
Resolves #45
Resolves #46

Tests are passing for many things but not everything.

Next PR will add tests for other core Step functionality. The PR after that will add tests for quilt packaging functionality. The final planned PR will add major documentation for everything.

[donovanr]

Resolves #33

[evamaxfield]

Notes for how I will be testing this:

[x] Run basic CLI for raw (run, push, checkout)
[x] Run workflow with raw (all run local)
[x] Run workflow with raw (all run distributed)

[evamaxfield]

Failed on distributed and assume would fail on deployed based off of the self.step_local_staging_dir path handling. We will need to store the relative path of self.step_local_staging_dir instead of absolute because it gets lost on the distributed worker.

Log:

(qt_distributed) jacksonb@n232:~/.aics_dask/logs/20200206_17:05:08$ cat dask-worker-210373.out
[2020-02-07 01:05:57,445] INFO - prefect.TaskRunner | Task 'Raw': Starting task run...
[2020-02-07 01:05:57,446] ERROR - prefect.TaskRunner | Unexpected error: FileNotFoundError(2, 'No such file or directory')
Traceback (most recent call last):
  File "/home/jacksonb/miniconda3/envs/qt_distributed/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/home/jacksonb/miniconda3/envs/qt_distributed/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 873, in get_task_run_state
    self.task.run, timeout=self.task.timeout, **raw_inputs
  File "/home/jacksonb/miniconda3/envs/qt_distributed/lib/python3.7/site-packages/prefect/utilities/executors.py", line 179, in timeout_handler
    return fn(*args, **kwargs)
  File "/allen/aics/modeling/jacksonb/projects/datastep/datastep/step.py", line 35, in wrapper
    with open(parameter_store, "w") as write_out:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/jacksonb/Desktop/active/cell/quilt_task/local_staging/raw/run_parameters.json'
[2020-02-07 01:05:57,454] INFO - prefect.TaskRunner | Task 'Raw': finished task run for task with final state: 'Failed'
[2020-02-07 01:05:57,468] INFO - prefect.TaskRunner | Task 'print_path': Starting task run...
[2020-02-07 01:05:57,468] ERROR - prefect.TaskRunner | Task 'print_path': unexpected error while running task: TypeError("'FileNotFoundError' object is not subscriptable")
Traceback (most recent call last):
  File "/home/jacksonb/miniconda3/envs/qt_distributed/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 260, in run
    executor=executor,
  File "/home/jacksonb/miniconda3/envs/qt_distributed/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 721, in run_mapped_task
    upstream_state.result[i],
TypeError: 'FileNotFoundError' object is not subscriptable
[2020-02-07 01:05:57,475] INFO - prefect.TaskRunner | Task 'print_hello': Starting task run...
[2020-02-07 01:05:57,478] INFO - prefect.TaskRunner | Task 'print_path': finished task run for task with final state: 'Failed'
[2020-02-07 01:05:57,478] ERROR - prefect.TaskRunner | Task 'print_hello': unexpected error while running task: TypeError("'FileNotFoundError' object is not subscriptable")
Traceback (most recent call last):
  File "/home/jacksonb/miniconda3/envs/qt_distributed/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 260, in run
    executor=executor,
  File "/home/jacksonb/miniconda3/envs/qt_distributed/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 721, in run_mapped_task
    upstream_state.result[i],
TypeError: 'FileNotFoundError' object is not subscriptable
[2020-02-07 01:05:57,481] INFO - prefect.TaskRunner | Task 'print_hello': finished task run for task with final state: 'Failed'

[donovanr]

yeah, that makes sense. i thin it might be worth taking a minute to think about hwo we want to handle paths in general -- what's absolute, what's local, what's local relative to what, etc.

i think there needs to be a project_local_staging_dir that's more or less absolute. that might be the only one that needs to be absolute (or able to be made absolute) -- i think everything else can be relative to that?

step_local_staging_dir can be relative to project_local_staging_dir -- i don't think that conceptually breaks anything? definitely some code that needs to be changed though.

then there's paths in the manifests, and i think for that we need some sort of convention. like, do we default to thinking of them as relative, and make them absolute on demand? or vice versa? or something else?

absolute paths to me are useful for actual file manipulation, while relative paths are useful for conceptual data organization. so since this project is about data organization, i think the first class path mode should be relative. e.g. any saving out of a manifest should give relative paths. then we have a convenience methods that makes them absolute on demand for using them? maybe?