Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Distributed.jl - possibility to use other Serialization libraries? #1

Open
schlichtanders opened this issue Oct 10, 2023 · 6 comments

Comments

@schlichtanders
Copy link

schlichtanders commented Oct 10, 2023

Hi,
since recently Pluto can run Distributed, however it does not work out of the box because the standard serialization is known not to work in Pluto, but Distributed relies on it (I couldn't find a way to change this)

The PlutoSerialization is maintained in this repo https://github.com/MichaelHatherly/PlutoSerialization.jl

It would be great to be able to use this Serialization instead of the default Serialization when using Distributed inside a Pluto notebook.

@jpsamaroo
Copy link
Member

ClusterSerializer could have a hook for customizing name binding lookups. I'm not sure what such a design would look like in the face of multiple libraries wanting to hook this functionality. Probably a hook could be defined per-Module, so that this hook will only be called on modules that Pluto creates.

@vchuravy vchuravy transferred this issue from JuliaLang/julia Oct 10, 2023
@schlichtanders
Copy link
Author

schlichtanders commented Oct 11, 2023

Looking into which reasoning was taken for the global random number generator it seems that the preferred route is to make the ClusterSerializer pluggable. JuliaLang/julia#23199 (comment)

These are the two places where the ClusterSerializer is constructed

So I guess we would need:

  • an AbstractClusterSerializer type for clarity
  • and make the above two places (and respective wrappers around it) somehow being able to pass an AbstractClusterSerializer object (similar to how rand and other methods pass a RNG object)

@schlichtanders
Copy link
Author

@MichaelHatherly you build PlutoSerialization.jl, probably you have some valuable insights what could be potential solutions/problems

@MichaelHatherly
Copy link
Member

Thanks for pointing this out @schlichtanders. I've got no extra insights at the moment since I've not had a chance yet to look into how Malt changes things now that Pluto is using it instead. If I get a chance I'll update this thread.

@jpsamaroo
Copy link
Member

Malt shouldn't be relevant here since this issue is relating to running code under Distributed directly (IIRC Pluto only runs its own internal code under Malt)

@schlichtanders
Copy link
Author

I would really love to see Distributed be fully supported inside Pluto and would like to try out things myself. (e.g. creating the pluggable ClusterSerializer interface thought about above)

@jpsamaroo do you know whether and how Distributed.jl could already be replace by a fork or the like?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants