Modular Installation

[william-silversmith]

@ceesem suggested we should allow more modular install of CloudVolume.

This could be a good idea, but it would also make it more complex for people to get started (assuming installation proceeds smoothly the first time). It also would be breaking to make people start to have to add cloud-volume[all] to an existing process. Does Python support adding qualifiers that remove dependencies? cloud-volume[gs] for example could negate S3 support unless cloud-volume[gs,s3] was specified.

[ceesem]

Unfortunately, there's no mechanism to do this in python (I've seen it called a "negative extra_requires", and it gets discussed every so often in various issues online). The pattern I think I prefer is to not install most/any formats and have try/except catches on each with an informational exception (i.e. something that tells you want to install to do what you're trying to do) if you try to use a format.

If you're releasing it as a major version change, I guess I don't see how a very easy-to-fix breaking change would be outside of standard expectations. Anyone running cloudvolume on a server should be pinning versions and for anyone else the errors should make it fixable within a few minutes. It doesn't sound more onerous than debugging the kind of issues linked to above that come up for most users at some point or another, and it makes sure that you aren't on the hook for any and all install issues that arise across a whole community of compression formats.

[william-silversmith]

This is a reasonable solution, but people have been generally highly critical of breaking changes. It's something I'm willing to consider but probably with lots of forewarning.

I think the PyChunkGraph group has been the most vocal about backwards compatibility, so if it's okay with you guys I'm much more open to it, but there are also other people using it too that have less engineering discipline with regards to pinning (which doesn't cover dependencies unless you do pip freeze and so can cause additional problems).

[ceesem]

Totally fair! I think we've gotten a lot better about pinning versions these days and basically do a pip freeze now.

Edited to add: I think there's a big difference between client software and server software in this respect. Strong pinning doesn't make much sense to do in an environment where you're doing exploratory analysis, but for stable server deployment having fixed versions has been a really useful change. I'll bring this up with PCG folks next week.

Also, the only reason I'm harping on this now is that we're getting ready for some tutorials where a number of people will install cloudvolume and I'm trying to get ahead of the install troubleshooting we might have to do.

[william-silversmith]

Makes sense. Here are some pathways toward a more modular CV:

slow) add support for different extras, warn users about a breaking change for a while, then switch it over

fast) change the pip install name or add a second name that supports modularity