Tutorial part 4

[franknoe]

Refers to:
https://github.com/markovmodel/adaptivemd/blob/master/examples/tutorial/4_example_advanced_tasks.ipynb
But addresses also general package/API issues along the way.

I'll have a number of questions / comments about the details of this notebook, but first of all I'm missing the overall picture that I think should be given here, so let me just start by a general question:

General:

• My understanding of a task is that it is run by the worker, so it can access files from the "staging area" (that I think is just the project storage, correct?), copy or link them to the worker directory, run things there (e.g. MD trajectory or PyEMMA analysis). Now the main task of the adaptive sampling package is to establish a communication between the workers and the decision-maker (strategy or 'brain'). My understanding is that this is done via the database, i.e. workers need to write information (in the form of models, or some compressed trajectory statistics, or other metadata) into the database, while the strategy needs to retrieve this information from the database in order to make the decision what to do next. The information how to do this is completely lacking so far. Can you write the notebooks - or alternatively the doc pages - with this in mind? I think then we can go more into details.
• I really like the way you solved file transfer on the task level, this looks very powerful and general.

Details (incomplete):

• There are still references to RP in this notebook. I guess these are out of date.
• sometimes there are three slashes in the filenames ///, not sure if this intended and what it means
• There's this project.trajectories.one that I have no clue what it means.
• Lines such as Transfer('file://{}/my_rpc_function.py' > 'worker://my_rpc_function.py). I'm not sure what this is - some meta-language that is parsed and translated into an actual bash command? I'm not sure why not just use bash at this stage, because other lines are just bash (like the echo example). Even in a meta-language, why use a > to denote copy direction, if you want to make it look like a function call, I'd rather use a comma separating the two arguments, otherwise it looks like a pipe.