Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Publish a platform-specific wheel for Apple M1 (macOS ARM64) on PyPI #207

Open
oliverl-21 opened this issue May 8, 2021 · 20 comments
Open
Labels
documentation Improvements or additions to documentation enhancement New feature or request

Comments

@oliverl-21
Copy link

oliverl-21 commented May 8, 2021

SUMMARY

add Package for macos arm64

ISSUE TYPE
  • Feature Idea
ADDITIONAL INFORMATION

N/A

@webknjaz webknjaz added the enhancement New feature or request label May 8, 2021
@webknjaz webknjaz changed the title Apple M1 Package Publish a platform-specific wheel for Apple M1 (macOS ARM64) on PyPI May 9, 2021
@webknjaz
Copy link
Member

webknjaz commented May 9, 2021

It's an interesting idea but until GitHub adds respective VMs to GHA, there's nothing we can do about this. So for now the users will have to install ansible-pylibssh from the sdist.

Refs:

@radokristof
Copy link

Well for me even building it locally does not work.
Before Ventura I was able to build it and install it locally. Now this fails with an error related to clang:

* Installing packages in isolated environment... (wheel)
* Building wheel...
[1/1] Cythonizing /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/pylibsshext/_libssh_version.pyx
running build_ext
building 'pylibsshext._libssh_version' extension
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src
creating /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/pylibsshext
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk -DCYTHON_TRACE=1 -DCYTHON_TRACE_NOGIL=1 -I/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-env-yn3ws_5a/include -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/pylibsshext/_libssh_version.c -o /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/pylibsshext/_libssh_version.o
creating build
creating build/lib.macosx-13-arm64-3.10
creating build/lib.macosx-13-arm64-3.10/pylibsshext
clang -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk -lssh -DCYTHON_TRACE=1 -DCYTHON_TRACE_NOGIL=1 /private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/tmpngpxapqx/private/var/folders/6l/48k83hdn30bc24655lbtzgx80000gn/T/build-via-sdist-xbvujv80/ansible-pylibssh-1.0.1a1/src/pylibsshext/_libssh_version.o -o build/lib.macosx-13-arm64-3.10/pylibsshext/_libssh_version.cpython-310-darwin.so
ld: library not found for -lssh
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command '/usr/bin/clang' failed with exit code 1

ERROR Backend subprocess exited when trying to invoke build_wheel
ERROR: InvocationError for command /Users/radokristof/Documents/git/pylibssh/.tox/build-dists/bin/python -m build --outdir /Users/radokristof/Documents/git/pylibssh/dist/ . (exited with code 1)

@l3ender
Copy link

l3ender commented Feb 1, 2023

Seeing the same error after installing libssh (via brew) and soft linking libssh into command line tools (ref). I see the comment to "install ansible-pylibssh from the sdist"; can anyone provide detail on it?

I am only trying to install ansible-pylibssh because it a dependency of the ansible.netcommon collection. :(

@dmacduff
Copy link

@l3ender @radokristof I was able to build locally with the correct compiler flags exported, no symlink required. Procedure.

@webknjaz
Copy link
Member

@dmacduff you shouldn't need to go through the contributor path (using tox etc.)
If you have all the build deps setting the env vars should be enough.
I'd expect CFLAGS="-I $(brew --prefix)/include -I ext -L $(brew --prefix)/lib -lssh" pip install ansible-pylibssh to just work. If you want to produce a wheel artifact, change the command to pip wheel or python -m build --wheel.

@dmacduff
Copy link

@webknjaz Brilliant! I spun up a new venv, and once I pip cache purged, your solution worked like a charm, no git clone, tox, or locally built wheels required. Much simpler. Now back to S.O. to update my answers.

@andresfrancopt
Copy link

Hi @webknjaz,
Will the solution you provided still work if I'm using a virtual environment?
Thanks in advance.

@webknjaz
Copy link
Member

@andresfrancopt the person commenting before you said it worked.

@samdoran
Copy link

There is actually a universal wheel available, but the dynamic libraries in the universal wheel are x86_64 only.

> pip download ansible-pylibssh
> unzip ansible_pylibssh-1.1.0-cp311-cp311-macosx_10_9_universal2.whl
> file pylibsshext/.dylibs/*
pylibsshext/.dylibs/libcrypto.1.1.dylib: Mach-O 64-bit dynamically linked shared library x86_64
pylibsshext/.dylibs/libssh.4.9.4.dylib:  Mach-O 64-bit dynamically linked shared library x86_64

The .so files in the wheel are universal.

So long as libssh is compiled and in the include and and linker paths (see command above), the --no-binary flag will force a build. This is essentially the same as building a wheel from source and installing it.

pip install --no-binary :all: ansible-pylibssh

@samdoran
Copy link

This was the root cause of [WARNING]: ansible-pylibssh not installed, falling back to paramiko being displayed even though ansible-pylibssh is installed.

The network connection plugin checks for pylibssh by running three imports. The second fails:

> python -c 'from pylibsshext.errors import LibsshSCPException, LibsshSessionException'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: dlopen(/Users/sdoran/.pyenv/versions/debug/lib/python3.11/site-packages/pylibsshext/errors.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_ssh_get_error'

@NilashishC
Copy link
Contributor

NilashishC commented Nov 2, 2023

Hi @ryanmerolle. Does the steps shared by @webknjaz or @samdoran their comment work for you?

@ryanmerolle
Copy link

Sorry I missed this update. Here is the error I get on my m1 MacBook:

$ pip install --no-binary :all: ansible-pylibssh
WARNING: Skipping /opt/homebrew/lib/python3.11/site-packages/cffi-1.16.0.dist-info due to invalid metadata entry 'name'
Collecting ansible-pylibssh
  Downloading ansible-pylibssh-1.1.0.tar.gz (106 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 106.2/106.2 kB 4.0 MB/s eta 0:00:00
  Installing build dependencies ... /
done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: ansible-pylibssh
  Building wheel for ansible-pylibssh (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for ansible-pylibssh (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      [1/1] Cythonizing /private/var/folders/ct/kd8qq0sd7y1218smxjw9fwsh0000gn/T/pip-install-gmxpv3cu/ansible-pylibssh_35f12a538de049858f6ca36822265f2e/src/pylibsshext/_libssh_version.pyx
      /private/var/folders/ct/kd8qq0sd7y1218smxjw9fwsh0000gn/T/pip-install-gmxpv3cu/ansible-pylibssh_35f12a538de049858f6ca36822265f2e/src/pylibsshext/_libssh_version.c:1201:10: fatal error: 'libssh/libssh.h' file not found
      #include "libssh/libssh.h"
               ^~~~~~~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for ansible-pylibssh
Failed to build ansible-pylibssh

@samdoran
Copy link

samdoran commented Nov 6, 2023

That looks like you are missing the path to libssh in your LDFLAGS and CPPFLAGS.

@ryanmerolle
Copy link

ryanmerolle commented Nov 22, 2023

It all makes sense to me, just not super intuitive on the mac/arm space.

If anyone stumbles on this, make sure to install libssh using brew if installing native to mac, or if using a linux container on mac install the relevant libssh-dev package.

Feel free to close this. Thanks for your help!

@webknjaz webknjaz added the documentation Improvements or additions to documentation label Nov 24, 2023
@webknjaz
Copy link
Member

I'm labeling this as a "docs" issue since it seems to me that this information would be useful to be documented. Especially in the context of Python 3.12 which we don't currently ship wheels for.

@ssbarnea
Copy link
Member

@webknjaz Cannot we really something to make this build out of the box? This bug is directly affecting devtools team and lightspeed work. Ideally shipping py312 arm64 wheel should even avoid the need to build it.

@webknjaz
Copy link
Member

There's no usable way of having runners for this. Maybe, it's possible to have universal2 macOS wheels but that would be something that's not even verified to function. I'll get back to this once I complete my other packaging related activities that may provide me with extra insights.

ssbarnea added a commit to ansible/ansible-language-server that referenced this issue Dec 11, 2023
- discover broken virtualens (after python upgrade)
- implement workaround for building ansible-pylibssh

Related: ansible/pylibssh#207
ssbarnea added a commit to ansible/ansible-language-server that referenced this issue Jan 21, 2024
- discover broken virtualens (after python upgrade)
- implement workaround for building ansible-pylibssh

Related: ansible/pylibssh#207
ssbarnea added a commit to ansible/ansible-language-server that referenced this issue Jan 21, 2024
* Improve test-setup.sh for macos

- discover broken virtualens (after python upgrade)
- implement workaround for building ansible-pylibssh

Related: ansible/pylibssh#207

* Make dockerfile dependabot compatible

See dependabot/dependabot-core#6067
@valiac
Copy link

valiac commented May 14, 2024

just adding my experience with it here:
with libssh 0.10.6 & libssh2 1.11.0_1 installed, the pip package installed without issues, but ansible still didn't detect it, even after re-installing with the env vars provided here by @webknjaz

The only way i could get it to work was locally building it as per https://stackoverflow.com/questions/74048675/cannot-install-ansible-pylibssh-on-macos

Afterwards ansible picked it up just fine

@webknjaz
Copy link
Member

libssh2 is separate from libssh and we don't use it so its presence doesn't matter.
libssh dev headers are needed to compile the C-extensions, when building from source. And so the env vars can tell the compiler where to find those headers. Once built, the wheel will be cached and the following pip installs will just unarchive it, provided that the Python version is the same.
It is possible that ansible-core didn't detect this project if you installed it into a different (virtual)env, not the one having Ansible installed. Different envs are isolated so the installed packages don't leak into one another.

@lastorel
Copy link

After checking multiple workarounds I find out - my ansible installed globally doesn't see any lib in venv like Святослав said
So at the moment if you have libssh installed by brew:

  1. NOT in venv: CFLAGS="-I $(brew --prefix)/include -I ext -L $(brew --prefix)/lib -lssh" pip install ansible-pylibssh

done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
None yet
Development

No branches or pull requests