Graceful Worker Process exit on restart

[philipstarkey]

Original report (archived issue) by Jan Werkmann (Bitbucket: PhyNerd, GitHub: PhyNerd).

---

Currently if one restarts a Tab the worker process is immediately terminated. This can leave things in a bad state. In particular I've been having problems with locks not being released. It would be nice if there was a attempt to call the shutdown() function of the workers before terminating them.

I already tried to implement this on a per worker basis with python's signal library, but under windows there is no signal.SIGTERM so that didn't work.

[philipstarkey]

Original comment by Philip Starkey (Bitbucket: pstarkey, GitHub: philipstarkey).

---

It should be possible to do this in zprocess I think. I recently did some work on another project (autoscrub) which relies on managing and killing subprocesses (although not necessarily killing them cleanly).

Things that I've found include:

• Using creationflags=subprocess.CREATE_NEW_PROCESS_GROUP in the call to Popen. zprocess doesn't seem to do this (interestingly there is a zprocess function which calls Popen such that the subprocess is detached, but then the subprocess_with_queues function doesn't use it. I'm not all that familiar with this part of zprocess, so can't comment on if there is a reason for that.
• You should be able to at least send a signal.SIGINT I think (although possibly implementing the above would stop that...)?
• Was this you? Seems like maybe it's a solution if you hadn't seen it...it sends a slightly different signal (not sure if it is cross platform or not, but that can be worked around).
• The Python package psutil is amazing and makes things a lot easier. We should consider using it instead of subprocess in zprocess.

Anyway, I would suggest we turn this into a zprocess issue as I think it can probably be mainly solved on that side, and then we can update BLACS to use the improved zprocess API.

[philipstarkey]

Original comment by Jan Werkmann (Bitbucket: PhyNerd, GitHub: PhyNerd).

---

• I'm not sure we really want to detach the process from the main process as that would make things a lot messier with them not closing on exit of the main process.
• I haven't tried sending any designated signals. I didn't want to touch zprocess as the shutdown routine is a BLACS thing and not something zprocess provides.
• No that wasn't me but the solution sounds promising I'll test this on Monday.
• Not sure if psutil would be much help in this case as terminate() maps to kill() which uses the windows api's TerminateProces just like in subprocess.

Your probably right that solving this in zprocess is a better solution. I'll open a issue there. I think we should keep this issue open anyway as others might run into the same problems I had.