Switch to "pyproject.toml" and src layout: Installing from source is not easy on WSL Linux and MacOS #336
Comments
|
Thanks for the detailed information here! I see two problems here, the first will be reliable fixed by moving to a
what method did you use to obtain the source for your installation? |
|
Thank you very much for your speedy reply! For the example above, I downloaded the package from main repository by clicking on Since I have already forked $ git clone [email protected]:omid-b/discretize
Cloning into 'discretize'...
remote: Enumerating objects: 34285, done.
remote: Counting objects: 100% (3976/3976), done.
remote: Compressing objects: 100% (1077/1077), done.
remote: Total 34285 (delta 2964), reused 3835 (delta 2879), pack-reused 30309
Receiving objects: 100% (34285/34285), 54.92 MiB | 514.00 KiB/s, done.
Resolving deltas: 100% (24698/24698), done.
$ cd discretize/Create a fresh env ... $ virtualenv env --python=3.10 && source env/bin/activate
created virtual environment CPython3.10.12.final.0-64 in 3058ms
creator CPython3Posix(dest=/mnt/c/Users/baghom/Desktop/Tests/env, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/omid-b/.local/share/virtualenv)
added seed packages: pip==22.0.2, setuptools==59.6.0, wheel==0.37.1
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivatorNow install ... $ pip install .
Processing /mnt/c/Users/baghom/Desktop/Tests/discretize
Preparing metadata (setup.py) ... done
Collecting cython>=0.29.31
Using cached Cython-3.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)
Collecting numpy>=1.20
Using cached numpy-1.25.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)
Collecting scipy>=1.8
Using cached scipy-1.11.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.3 MB)
Collecting setuptools_scm
Using cached setuptools_scm-7.1.0-py3-none-any.whl (43 kB)
Collecting tomli>=1.0.0
Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting packaging>=20.0
Using cached packaging-23.1-py3-none-any.whl (48 kB)
Collecting typing-extensions
Using cached typing_extensions-4.7.1-py3-none-any.whl (33 kB)
Requirement already satisfied: setuptools in /mnt/c/Users/baghom/Desktop/Tests/env/lib/python3.10/site-packages (from setuptools_scm->discretize==0.1.dev5527+g58043b9) (59.6.0)
Building wheels for collected packages: discretize
Building wheel for discretize (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [6 lines of output]
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/mnt/c/Users/baghom/Desktop/Tests/discretize/setup.py", line 80, in <module>
from Cython.Build import cythonize
ModuleNotFoundError: No module named 'Cython'
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for discretize
Running setup.py clean for discretize
Failed to build discretize
Installing collected packages: typing-extensions, tomli, packaging, numpy, cython, setuptools_scm, scipy, discretize |
$ pip install cython
Requirement already satisfied: cython in /mnt/c/Users/baghom/Desktop/Tests/env/lib/python3.10/site-packages (3.0.2)This is weird! |
|
Following #332 , I downgraded my cython $ pip install cython==0.29.35
Collecting cython==0.29.35
Using cached Cython-0.29.35-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (1.9 MB)
Installing collected packages: cython
Attempting uninstall: cython
Found existing installation: Cython 3.0.2
Uninstalling Cython-3.0.2:
Successfully uninstalled Cython-3.0.2
Successfully installed cython-0.29.35try again... $ pip install .
Processing /mnt/c/Users/baghom/Desktop/Tests/discretize
Preparing metadata (setup.py) ... done
Requirement already satisfied: cython>=0.29.31 in /mnt/c/Users/baghom/Desktop/Tests/env/lib/python3.10/site-packages (from discretize==0.1.dev5527+g58043b9) (0.29.35)
Requirement already satisfied: numpy>=1.20 in /mnt/c/Users/baghom/Desktop/Tests/env/lib/python3.10/site-packages (from discretize==0.1.dev5527+g58043b9) (1.25.2)
Requirement already satisfied: scipy>=1.8 in /mnt/c/Users/baghom/Desktop/Tests/env/lib/python3.10/site-packages (from discretize==0.1.dev5527+g58043b9) (1.11.2)
Requirement already satisfied: setuptools_scm in /mnt/c/Users/baghom/Desktop/Tests/env/lib/python3.10/site-packages (from discretize==0.1.dev5527+g58043b9) (7.1.0)
Requirement already satisfied: packaging>=20.0 in /mnt/c/Users/baghom/Desktop/Tests/env/lib/python3.10/site-packages (from setuptools_scm->discretize==0.1.dev5527+g58043b9) (23.1)
Requirement already satisfied: typing-extensions in /mnt/c/Users/baghom/Desktop/Tests/env/lib/python3.10/site-packages (from setuptools_scm->discretize==0.1.dev5527+g58043b9) (4.7.1)
Requirement already satisfied: setuptools in /mnt/c/Users/baghom/Desktop/Tests/env/lib/python3.10/site-packages (from setuptools_scm->discretize==0.1.dev5527+g58043b9) (59.6.0)
Requirement already satisfied: tomli>=1.0.0 in /mnt/c/Users/baghom/Desktop/Tests/env/lib/python3.10/site-packages (from setuptools_scm->discretize==0.1.dev5527+g58043b9) (2.0.1)
Building wheels for collected packages: discretize
Building wheel for discretize (setup.py) ... done
Created wheel for discretize: filename=discretize-0.1.dev5527+g58043b9-cp310-cp310-linux_x86_64.whl size=4691875 sha256=60e3e11dfbcc8d6bc25777bf9eda2704321ede39e456811fdcb553218c87acd3
Stored in directory: /tmp/pip-ephem-wheel-cache-0yplpgrx/wheels/fe/72/26/d10ab46336913418a55201c5aadc78a51d73715b4e0c9e47ca
Successfully built discretize
Installing collected packages: discretize
Attempting uninstall: discretize
Found existing installation: discretize 0.1.dev5527+g58043b9
Uninstalling discretize-0.1.dev5527+g58043b9:
Successfully uninstalled discretize-0.1.dev5527+g58043b9
Successfully installed discretize-0.1.dev5527+g58043b9Nice! Let's test it now! $ python -c "import discretize"
cannot import name 'interputils_cython' from 'discretize._extensions' (/mnt/c/Users/baghom/Desktop/Tests/discretize/discretize/_extensions/__init__.py)
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/mnt/c/Users/baghom/Desktop/Tests/discretize/discretize/__init__.py", line 42, in <module>
from discretize.unstructured_mesh import SimplexMesh
File "/mnt/c/Users/baghom/Desktop/Tests/discretize/discretize/unstructured_mesh.py", line 7, in <module>
from discretize._extensions.simplex_helpers import (
ModuleNotFoundError: No module named 'discretize._extensions.simplex_helpers'Trouble! it seemed like it was installed properly, but it did not build the extensions apparently! |
|
And if you change directories out of the source folders are you able to? |
|
Yes as you mentioned, it tries to read from the source folder rather than my env. Here is the hotfix (but I still insist installation of discretize should not involve this many troubles!) cd into source code dir and then copying the build files here... $ cd /mnt/c/Users/baghom/Desktop/Tests/discretize/discretize/_extensions/
$ cp ../../../env/lib/python3.10/site-packages/discretize/_extensions/*.so ./
$ python -c "import discretize"
$ python -c "import discretize._extensions"
$ python -c "import discretize.tests"No errors now... |
|
But if you're doing that, why not just do an editable install? |
|
Yes I could... but how about others who are not python experts (not that I am an expert; not at all!), and just want to install and use SimPEG/Discretize? I can make it fixed the if owners/main-developers accept my contribution... |
|
I'm just trying to make sure we understand the root cause of the issue that you are encountering, which is as I said, Cython, numpy, and setuptools_scm are required for building the package, but only numpy is a run-time requirement. The current setup.py configuration does not successfully install the needed files when this occurs. |
|
I understand your point, but still believe that it could be easily fixed to make the process more hassle-free... An easy fix perhaps would be to modify cython==0.29.35
numpy
setuptools_scm
-e .But this hotfix is very temporary and things will go wrong again soon. Moving toward a src-layout and |
In addition to making changes for src-layout, also fixed find_package() configurations, so both editable and non-editable installation now work as intended. Previousely, only editable installation was possible (simpeg#336).
Issue explanation
Trying to install discretize from source using the recommended/typical instructions does not work on Linux (tested on WSL linux) and MacOS. This is easily replicable when trying to install using the following methods even in a fresh Conda environment (installing discretize in a fresh python environment, using
virtualenvorvenv, is even more difficult):python setup.py install(suggested in current documentation)pip install .Before I demonstrate the issue, I create a fresh conda environment using the following commands:
conda create -p env python=3.10 && conda activate env/Example to show the issue
My first try...
Okay, let's install
cythonfirst...Try again ...
Okay, we need
numpytoo, andpython setup.py installcannot figure out the install dependencies!Try again...
Let's use the
pipmethod...Add flag
--use-pep517What if I try developer mode installation?
What if I first compile in-place ?
Could it be due to
setuptools_scmcannot figure out the program version? I modify the code to a manual version 0.0.0 to see if that helps!YES Finally! Now, let's attempt to install the package!
A simple test to see if discretize is importable from root directory:
Terrific! No errors!
Recommendation
I recommend switching
discretizeover topyproject.tomlandsrc-layoutconfiguration. In my experience, the current package layout and configuration is very limiting and definitely not future-proof....Following my discussion with @jcapriot on Slack, I will make 2 PRs that if merged, will make installation of discretize hassle-free even on a fresh python environment.
The text was updated successfully, but these errors were encountered: