provide binary wheels for Linux compatible with lxml

[nijel]

I know this error has been added intentionally to catch other breakages, but is there a way to provide binary wheels that will work with current lxml? So that pip install lxml xmlsec will bring working combo.

Presently, it doesn't:

(.venv) $ pip install lxml xmlsec
Collecting lxml
  Using cached lxml-5.2.1-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (3.4 kB)
Collecting xmlsec
  Downloading xmlsec-1.3.14.tar.gz (68 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.9/68.9 kB 549.7 kB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Using cached lxml-5.2.1-cp311-cp311-manylinux_2_28_x86_64.whl (5.0 MB)
Building wheels for collected packages: xmlsec
  Building wheel for xmlsec (pyproject.toml) ... done
  Created wheel for xmlsec: filename=xmlsec-1.3.14-cp311-cp311-linux_x86_64.whl size=160772 sha256=79e2ffc317a1754761d069e1e76adf033ac7be8e4189c816411284fe4eee6d41
  Stored in directory: /home/nijel/.cache/pip/wheels/98/60/8b/bc03288ec33436feed643c9a680f5124f697a735c7cb5f0d23
Successfully built xmlsec
Installing collected packages: lxml, xmlsec
Successfully installed lxml-5.2.1 xmlsec-1.3.14
(.venv) $ python -c 'import xmlsec'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
xmlsec.InternalError: (-1, 'lxml & xmlsec libxml2 library version mismatch')

The problem is that lxml wheels are built with internal libxml while xmlsec is using system one.

Rebuilding lxml against system libxml does address the issue, but working combo out of the box would be better.

[mxamin]

@nijel The workaround solution would be to force lxml not to use the pre-compiled wheel: pip install lxml xmlsec --no-binary=lxml
We already have pre-compile wheels for linux and macos (see release note: https://github.com/xmlsec/python-xmlsec/releases/tag/1.3.14) but I didn't upload them in pypi because of this version checking that was added in the latest release.
I'm gonna note this issue for the next release, meanwhile can you provide the following information:

• OS distribution and version
• xmlsec1 library version
• libxml2 version
• python version

[nijel]

I know the workaround, the problem is that there is no way to tell this inside dependencies. Instead, you need to tell users to not use pip install <package>, but pip install --no-binary=lxml <package>, what half of them won't do and complain that things do not work then :-).

I'm on Debian 12 (Python 3.11, libxml2 2.9.14, xmlsec1 1.2.37), it happens on GitHub CI environment with Ubuntu 22.04 (any Python version installed by actions/setup-python, libxml 2.9.13, xmlsec1 1.2.33).

PS: I'm happy user of https://github.com/pypa/cibuildwheel and https://github.com/pypa/gh-action-pypi-publish to automate all the wheel builds and publishing.

[mxamin]

@nijel You're right, I put it in the todo list for the next release. Hope we can solve it better :)

[jonathangreen]

@mxamin i like the idea of providing Linux wheels for the next release as well. We are already making the wheels with cibuildwheel, so it's just a matter of distributing them.

I can work on a PR to automate attaching the wheels to the release in GitHub and uploading them to PyPI if you would like.

[mxamin]

@mxamin i like the idea of providing Linux wheels for the next release as well. We are already making the wheels with cibuildwheel, so it's just a matter of distributing them.

I can work on a PR to automate attaching the wheels to the release in GitHub and uploading them to PyPI if you would like.

I could have do it manually (simply downloading the artificats and uploading them to pypi) but I decided not to do it because of libxml2 version check. Assume that lxml wheel is build with version 2.9.x and python-xmlsec with 2.11.x, we would end up with the mismatch error which is not good. So the real problem here is to fix the lxml/python-xmlsec compatibility issue for real :)

[jonathangreen]

Making the integration with lxml safer, like you are suggesting, would be a win for sure.

Even if we do that though, I think there is still a strong argument for providing wheels for Linux. It makes things much easier for end users of the package, especially when installing inside a container where you don't necessarily have all the build tooling available.

[mxamin]

Definitely it would help. I wonder if the created wheels would help in reported cases. I probably check that later.

[nijel]

I was thinking if it is possible to build a private xmlsec1 against libxml2 from lxml and ship it inside the wheel.

[jonathangreen]

@nijel would you be able to manually install one of the Linux wheels attached the the release in GitHub https://github.com/xmlsec/python-xmlsec/releases/tag/1.3.14 and see if it solves your issue?

The wheels are statically linked against libxml2 and libxmlsec, so I believe it will sort your issue. Those are the wheels we are discussing uploading to pypi.

[nijel]

Thanks, that seems to work locally just fine. Pushed that to CI now to see the outcome there.

[jonathangreen]

@nijel yesterday afternoon, after some testing, @mxamin uploaded these same wheels to pypi. You should just be able to install xmlsec normally in CI now.

See:
#315 (comment)

I think that satisfies this ticket.

[nijel]

Is it possible that after this change, I get the lxml & xmlsec libxml2 library version mismatch error with lxml built locally? The CI started to fail for us meanwhile, so this might be the reason.

Maybe the version check probably can be skipped in case xmlsec is linked statically with the libxml?

[jonathangreen]

Is it possible that after this change, I get the lxml & xmlsec libxml2 library version mismatch error with lxml built locally? The CI started to fail for us meanwhile, so this might be the reason.

It looks like maybe you were installing lxml with no-binary, but installing the binary wheel for xmlsec. Now that we have wheels uploaded both lxml and xmlsec need to be built the same way. Either installed from wheels or built with no-binary.

Maybe the version check probably can be skipped in case xmlsec is linked statically with the libxml?

Because xmlsec passes data into lxml both need to be built with the same version of libxml2. This situation is unfortunate, and we'd like to fix it, so this requirement doesn't exist. But for now it does.

This means that either they both need to be installed from wheels, which includes same version of libxml2 or they both need to be compiled using no-binary so they both get the system version of libxml2. If you mix and match these approaches you will see version mismatch error.

[nijel]

Thanks for explanation, it now works for me. Publishing binary wheels while I was testing this in CI made results for me quite unpredictable, but now with wheels-only setup, everything works just fine. Thanks for your effort to make this work!