Add threads for alias and output updates on startup

[Atticus1806]

Fix #213

[albertz]

How thread-safe is Sisyphus? There is no potential issue with that?

[Atticus1806]

Since this is just alias and output updates I dont think there is a problem. These ops should be executable any time

[albertz]

I'm more thinking about all the internal caching mechanisms of jobs, outputs, paths, the graph, etc. All of that must be thread-safe. (But it might be already, as we also use multi-threading elsewhere. I don't recall the details now.)

[Atticus1806]

While my impression is they are, I am passing the question to the experts since I dont know for sure.

[critias]

I don't see any problem running create_aliases in a separate thread, but check_output might be problematic. If it slows down the startup to much you could consider removing it here. It will be called again inside the manager loop anyway.
That should be tested, but I think that's less dangerous than running it in an extra thread.

Side note: It would be nicer to use the already existing manager thread_pool instead of starting new threads.

[Atticus1806]

I updated to use the thread pool.

It will be called again inside the manager loop anyway.

While that is true, its called with other parameters right? So a "full" update is only done at the beginning. I dont mind removing it here and then saying "user needs to call update for a full update", but this should be discussed.
Or do we add a flag Skip_Output_update_on_startup or smth.?
If we want to keep it but thread it: any suggestions on how to check if this actually has no unwanted behavior? Maybe we can call it only for non active outputs or smth?

[critias]

Good point, a simple solution might be to just add a semaphore at the start of check_output to ensure that it's not run twice in parallel.

[Atticus1806]

I am currently testing the semaphore version, but what I am wondering about is this line:

sisyphus/sisyphus/manager.py

Line 485 in d256f43

self.check_output(write_output=False, update_all_outputs=True)

This looks quite redundant too me, since it is called in all relevant paths later aswell. Maybe this can be removed here?