Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ordered shutdown hooks for dependent threadpools #46

Open
mourjo opened this issue Jun 27, 2020 · 3 comments
Open

Ordered shutdown hooks for dependent threadpools #46

mourjo opened this issue Jun 27, 2020 · 3 comments

Comments

@mourjo
Copy link

mourjo commented Jun 27, 2020

Hi! I wanted to ask, do you think it would be a good idea to have a lifecycle manager (a shutdown hook) for multiple claypoole threadpools for when the JVM gets a kill signal? Something like if threadpool t1 uses another threadpool t2, then at the JVM shutdown, we would want the threadpool t2 to shutdown first in order for t1 to be gracefully shut down without having tasks rejected? We could have one shutdown hook for all claypoole threads in a JVM and based on the dependency (which I guess needs to be mentioned at the time of creation of the pool), it could coordinate shutting down of the different pools. Do you think this would be something worth having in this project? I could start a pull request then. Wanted to get your opinion first.

https://clojureverse.org/t/is-there-a-canonical-way-or-framework-to-handle-threads-lifecycle/6136

@leon-barrett
Copy link
Contributor

Hi there! Sorry for the delay.

I like the idea of managed shutdown. In general, however, the dependency graph might be cyclic, e.g.:

(def http-pool (cp/threadpool))
(def db-pool (cp/threadpool))

(def results
  (->> items
    (cp/pmap http-pool http-request-1)
    (cp/pmap db-pool fetch-db-data)
    (cp/pmap http-pool http-request-2)))

Also, I do think something like https://github.com/stuartsierra/component is generally a better fit than exit-hook magic. The main advantage of Component is avoiding global mutable state, and I think that would be helpful here.

That said, if you have an API proposal, I'd be happy to consider it. The following considerations seem relevant:

  • I'd like to avoid making it a default, because that's too magical. I'd prefer an explicit function like (add-shutdown-hook my-threadpool :after other-threadpool).
  • I'd like to avoid dependencies on other libraries. In general, reuse is good, but I think threadpools are such a foundational element that I'd like to keep Claypoole dependency-free.

Another option that might make sense is just to take your graceful shutdown article https://medium.com/helpshift-engineering/achieving-graceful-restarts-of-clojure-services-b3a3b9c1d60d and make it a library that composes nicely with Claypoole. (It didn't look like it was a published library yet, but I wasn't sure.) The advantage there is that you can make something more general-purpose, and it might help meet that need for more cases than just Claypoole threadpools.

Thanks for the idea, and let me know how you want to follow up.

@mourjo
Copy link
Author

mourjo commented Jul 1, 2020

Hey thanks for taking the time to reply in detail! Let me chalk up something, I like the idea of an independent project which can play well with shutting down threadpools in general. Initially, I was also thinking of a DAG-like approach (like namespaces which can't depend on each other), so in that respect, the :after API sounds good). What I'll do is try to implement something like this in a separate repository and share it here maybe. Or maybe it's better to close this issue. Thanks for the help though, appreciate it!

@ieugen
Copy link
Member

ieugen commented Feb 27, 2022

Hi @mourjo ,

Do you think this issue is still relevant / something worth pursuing ?

Leon does not work with Clojure and we managed to move the project here (clj-commons) and I stepped in to maintain it.
The project is currently feature complete and we don't expect much change to happen besides maintenance and docs improvements.

I haven't had a chance to read through your proposal yet.
I decided to ask your thoughts on it first.
Please reach out with anything related to claypoole.

Thanks,
Eugen

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants