Python-modules site-package interferes with python virtualenv

[raymondEhlers]

Dear Experts,

I recently updated my alidist, and it appears that some changes in how python is handled have make my workflow unusable. I use separate virtual environments per python package which I develop (which as I understand it, is a best practice for development). But it appears that adding the alibuild site-packages to the PATH and PYTHONPATH makes using a virtualenv extremely difficult.

For example, since pytest is installed in the alibuild site-packages, it expects for all of my packages (and their dependencies) to be installed in that site-packages too (I assume because the pytest executable is picked up from the PATH). This means that I have to reinstall all of my development packages into the alibuild site-packages, effectively preventing me from using a virtualenv. I worked around this by leaving the virtualenv and removing the pytest package from the alibuild site-packages, but I don't think this is a good solution. Updating packages from the virtualenv requires targeting the alibuild site-packages (with pip install --target), but it's easy for packages to appear updated, only to be shadowed by the alibuild site-packages older version.

I understand that reproducible builds are needed on the grid, but I've found this experience extremely frustrating from a users perspective. Can something be done to improve this user experience? Can a flag be added to alibuild that tells it that it's in a virtualenv and pickup the packages from there?

Any help is greatly appreciated! Thanks!

[ktf]

#2034 might solve this, once merged.

[raymondEhlers]

Thanks! I'll try it out when I get a chance.

If I understand the changes correctly, it's definitely an improvement (I should be able to easily load the aliBuild created virtualenv to change packages if I must), but I don't think this will entirely solve my issue. Namely, I think the virtualenv created by aliBuild will still shadow any other virtualenv that I might load. So I still won't be able to update any packages that are stored in python-modules. The only way I see around this is if the user somehow is allowed to have their virtualenv override aliBuild python-module packages. But I suppose this could cause issues with packages that are built against python-modules

[ktf]

We checked this with @miranov25 today. What we really need to do on top is an alivirtualenv which does the following:

• Use it to create a new virtualenv
• Put python-modules $PYTHONPATH in a .pth file in the virtualenv site-packages folder.
• Change the activate script to do unset $PYTHONPATH.

This will allow a new virtualenv on top of the one which is provided with python-modules.

[raymondEhlers]

That sounds like it would be a good solution. I look forward to the future development. Thanks!