Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Graceful Worker Process exit on restart #31

Open
philipstarkey opened this issue Nov 17, 2017 · 2 comments
Open

Graceful Worker Process exit on restart #31

philipstarkey opened this issue Nov 17, 2017 · 2 comments
Labels
enhancement New feature or request major

Comments

@philipstarkey
Copy link
Member

Original report (archived issue) by Jan Werkmann (Bitbucket: PhyNerd, GitHub: PhyNerd).


Currently if one restarts a Tab the worker process is immediately terminated. This can leave things in a bad state. In particular I've been having problems with locks not being released. It would be nice if there was a attempt to call the shutdown() function of the workers before terminating them.

I already tried to implement this on a per worker basis with python's signal library, but under windows there is no signal.SIGTERM so that didn't work.

@philipstarkey
Copy link
Member Author

Original comment by Philip Starkey (Bitbucket: pstarkey, GitHub: philipstarkey).


It should be possible to do this in zprocess I think. I recently did some work on another project (autoscrub) which relies on managing and killing subprocesses (although not necessarily killing them cleanly).

Things that I've found include:

  • Using creationflags=subprocess.CREATE_NEW_PROCESS_GROUP in the call to Popen. zprocess doesn't seem to do this (interestingly there is a zprocess function which calls Popen such that the subprocess is detached, but then the subprocess_with_queues function doesn't use it. I'm not all that familiar with this part of zprocess, so can't comment on if there is a reason for that.
  • You should be able to at least send a signal.SIGINT I think (although possibly implementing the above would stop that...)?
  • Was this you? Seems like maybe it's a solution if you hadn't seen it...it sends a slightly different signal (not sure if it is cross platform or not, but that can be worked around).
  • The Python package psutil is amazing and makes things a lot easier. We should consider using it instead of subprocess in zprocess.

Anyway, I would suggest we turn this into a zprocess issue as I think it can probably be mainly solved on that side, and then we can update BLACS to use the improved zprocess API.

@philipstarkey
Copy link
Member Author

Original comment by Jan Werkmann (Bitbucket: PhyNerd, GitHub: PhyNerd).


  • I'm not sure we really want to detach the process from the main process as that would make things a lot messier with them not closing on exit of the main process.
  • I haven't tried sending any designated signals. I didn't want to touch zprocess as the shutdown routine is a BLACS thing and not something zprocess provides.
  • No that wasn't me but the solution sounds promising I'll test this on Monday.
  • Not sure if psutil would be much help in this case as terminate() maps to kill() which uses the windows api's TerminateProces just like in subprocess.

Your probably right that solving this in zprocess is a better solution. I'll open a issue there. I think we should keep this issue open anyway as others might run into the same problems I had.

@philipstarkey philipstarkey added major enhancement New feature or request labels Apr 5, 2020
Loki27182 pushed a commit to Loki27182/blacs that referenced this issue Oct 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request major
Projects
None yet
Development

No branches or pull requests

1 participant