Installing with python bridge support

[joellama]

I’m having some issues installing GDL with the python bridge. The documentation suggests to call with —enable-shared but I can’t figure out where to add that flag. If I try:

bash scripts/build_gdl.sh configure —enable-shared I get a usage error.

Apologies if I’ve missed something in the documentation, but I couldn’t see it anywhere. Any help would be really appreciated.

[slayoo]

Hello @joellama,
Are you intending to use GDL from within Python or Python from within GDL?
BTW, where is it suggested to used the "--enable-shared" option? (seems an outdated resource from before we had switched to CMake).
Thanks

[joellama]

Hey @slayoo,
I'm hoping to use GDL from within python. The thing I'm struggling with is how to compile so that I can type import GDL within python.

The enable-shared is mentioned on line 281 of https://github.com/gnudatalanguage/gdl/blob/master/README

As a hack though I have GDL running through os.system which has enabled me to parallelize my code! I'd still love to know how to get the GDL package compiled for use though within python. Thanks again!

[slayoo]

after downloading GDL source, try something along these lines:

cd gdl
mkdir build
cd build
cmake -DPYTHON_MODULE=ON .. 
make -j
PYTHONPATH=$PYTHONPATH:`find . -name GDL.so | xargs dirname` python -m pytest ../testsuite/python/test-GDL.py 

The last line executes the Python-written tests for the Python GDL module (the one imported by import GDL)
HTH

[joellama]

@slayoo, thanks for your help with this. I got a lot closer. With the commands you listed above, I now have GDL.so; however, I'm having trouble with import GDL where I get the error
ImportError: dlopen(/Users/joellama/gdl/build/src/GDL.so, 2): Library not loaded: @rpath/libantlr.dylib Referenced from: /Users/joellama/gdl/build/src/GDL.so Reason: image not found

In the build/src/ directory (which is where GDL.so is), I can see antlr/libantlr.dylib but I'm not sure how to link to it. Apologies for my basic questions with this.

[slayoo]

Please try adding the directory in which libantlr.dylib is located to the list of paths in DYLD_LIBRARY_PATH or DYLD_FALLBACK_LIBRARY_PATH env vars; see man dyld.

Also, perhaps doing sudo make install will help as it should try to install the Python module within Python's site directory (https://github.com/gnudatalanguage/gdl/blob/master/src/CMakeLists.txt#L243).

HTH

[mlangelaar]

Good morning, I am having what might be a related issue ??? If you would like me to post as a new
issue instead (the term hijack a thread seems to common lately, I apologize if so), then please let me
know and I will do so (deleting this comment as well).

I ran into this thread after the grad student I am helping actually found it ...

EDIT -> I ran into the antlr issue weeks ago, got past that, it's just the PyInit_GDL now :(
(perhaps I should post / move this to NEW issue)

1. import GDL on my system says dynamic module missing the module export function PyInit_GDL

2. I decided to use the build and test from Dec 17, 2021 that Sylwester asked the person to do.
   (in my case PYTHON=OFF, so I just went with what he has)

   Using the Python-written test just now, interestingly enough I am seeing an undefined symbol
   as shown by the test results below :

============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /root/GDL/gdl
plugins: filter-subpackage-0.1.1, openfiles-0.5.0, astropy-header-0.1.2, arraydiff-0.3, hypothesis-6.14.3, doctestplus-0.10.0, remotedata-0.3.2, cov-2.12.1
collected 0 items / 1 error

==================================== ERRORS ====================================
________________ ERROR collecting testsuite/python/test-GDL.py _________________
ImportError while importing test module '/root/GDL/gdl/testsuite/python/test-GDL.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/rh/rh-python38/root/usr/lib64/python3.8/importlib/init.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
../testsuite/python/test-GDL.py:12: in
import GDL
E ImportError: /root/GDL/gdl/build/src/GDL.so: undefined symbol: _Z8wxGetAppv
=========================== short test summary info ============================
ERROR ../testsuite/python/test-GDL.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 1.09s ===============================

Maiko

[slayoo]

CC @GillesDuvert

Seems, the compiled binary has some troubles in using wxGetApp which is part of interactive plotting / widgets machinery that is anyhow likely unusable from within the Python bridge (?). I'd suggest to try compiling without wx support by invoking cmake with

 -DWXWIDGETS=OFF -DINTERACTIVE_GRAPHICS=OFF 

[mlangelaar]

I tried that already, but I was using -DPYTHON=OFF, so I did it again just now, without that define.

This time the Python-written script is saying exactly what the import GDL is saying :

Traceback:
/opt/rh/rh-python38/root/usr/lib64/python3.8/importlib/init.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
../testsuite/python/test-GDL.py:12: in
import GDL
E ImportError: dynamic module does not define module export function (PyInit_GDL)

Maiko

[slayoo]

Just to confirm, does this help: cmake -DWXWIDGETS=OFF -DINTERACTIVE_GRAPHICS=OFF -DPYTHON=OFF?

[mlangelaar]

No, searching for PyInit_GDL surprisingly (?) comes up with nothing online.
I've spent hours on this, various combos with cmake, starting to wonder if
this is devtool related, so I may find another server and use stock OS python
and so on, the student will just have to wait ...
EDIT(S) -> there was a reason to use devtools actually, stock GCC fails compile,
and we needed a more recent version of Python (3), anyways ...

[slayoo]

Would it be possible for you to try the Debian package of the GDL Python module: https://packages.debian.org/sid/python3-gdl

[mlangelaar]

I can do that, will try it when I'm back at work, thank you.

…

On 2023-02-19 2:58 p.m., Sylwester Arabas wrote: Would it be possible for you to try the Debian package of the GDL Python module: https://packages.debian.org/sid/python3-gdl <https://packages.debian.org/sid/python3-gdl> — Reply to this email directly, view it on GitHub <#1231 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACFGMEZ5IWMYMIBT5G4XYLWYKCPTANCNFSM5KJWRTXQ>. You are receiving this because you commented.Message ID: ***@***.***>

[GillesDuvert]

Indeed, wxGetApp() is undefined since it is created in the main (gdl.cpp) program and not in a library. It is thus absent from the GDL.so
The best workaround is not absolutely straightforward...

[GillesDuvert]

In other terms, as long as -DWXWIDGETS=ON Python will not work.

[GillesDuvert]

solved in #1520
pytest reports 10 failures and 49 passed tests...

[slayoo]

Thank you @GillesDuvert! I'm also not familiar with the Python bridge, I've never used it besides running the pytest tests that @olebole contributed.

I was not even able to find the error report that must have been created by pytest smile

Since all tests have passed, no report was issued.

[GillesDuvert]

I see 49 passed, 10 xfailed in 1.07s in the output?

[slayoo]

xfail means expected failure (e.g., a test that checks if an exception is raised for an out-of-bounds array access) - all OK!

[mlangelaar]

Quoting Gilles :

But for the rest, I just followed @slayoo 's commands:

cmake -DPYTHON_MODULE=ON ..
make -j
PYTHONPATH=$PYTHONPATH:find . -name GDL.so | xargs dirname python -m pytest ../testsuite/python ........
[snip]

Believe it or not, it is still failing, not able to find the PyInit_GDL

I don't want to waste your time anymore on this, it could be a variety of reasons, that I will investigate
on my own time I guess. I should be clear, the minimal cmake I am using is this, since I don't have QHULL
installed, and needed a self compiled version of HDF5 :

cmake3 -DHDF5DIR=/usr/local/hdf5 -DQHULL=OFF -DPYTHON_MODULE=ON ..

The only other 'questionable' thing I noticed during the cmake is this :

 -- Found wxWidgets: -pthread;;;-lwx_baseu-3.0;-lwx_gtk3u_core-3.0;-lwx_gtk3u_adv-3.0 (found version "3.0.4")
 -- INFO: We prefer to use GraphicsMagick than ImageMagick

I have installed GraphicsMagick in past attempts with no change in results.

I am using this environment, since the OS versions are 'too old' for this :

scl enable devtoolset-9 rh-python38 rh-perl530 bash

And in all fairness, some modules errored out on me during compile, to which I did
a simple 'uncomment' to move forward, these were less then minor errors, suggesting
that perhaps I am not meeting the proper prerequistes for packages needed to do this.

I will try (at Sylwesters suggestion) that Debian package he mentioned.

Thanks to everyone for trying, I will let you know how it goes when I have time to look at this again.

Maiko Langelaar / IT Department of Physics and Astronomy
University of Manitoba

[GillesDuvert]

FYI the cmake messages above are OK, although the 'preference' to GraphicsMagick is largely a phantom of the past.

[slayoo]

Googling the "dynamic module does not define module export function" message, it seems that in some cases this is related with Python version mismatch (the one that GDL module was compiled against vs. the one that is used when importing).

In principle, there are three Python versions that must match:

• the one CMake finds with find_package(PythonInterp)
• the one CMake finds with find_package(PythonLibs)
• the one used at runtime

In the Debian Python module package source ("rules" file), they specify the following CMake options for building the module:

	  -DPYTHON=YES -DPYTHONVERSION=3 -DPYTHON_MODULE=YES

perhaps you could try with setting PYTHONVERSION=3.8.6 to make sure it is consistent?

[mlangelaar]

This is VERY helpful information, thanks Sylwester, will work with this.

…

On 2023-02-25 03:36, Sylwester Arabas wrote: Googling the "dynamic module does not define module export function" message, it seems that in some cases this is related with Python version mismatch (the one that GDL module was compiled against vs. the one that is used when importing). In principle, there are three Python versions that must match: * the one CMake finds with find_package(PythonInterp) * the one CMake finds with find_package(PythonLibs) * the one used at runtime In the Debian Python module package source ("rules" file), they specify the following CMake options for building the module: -DPYTHON=YES -DPYTHONVERSION=3 -DPYTHON_MODULE=YES perhaps you could try with setting PYTHONVERSION=3.8.6 to make sure it is consistent? -- Reply to this email directly, view it on GitHub [1], or unsubscribe [2]. You are receiving this because you commented.Message ID: ***@***.***> Links: ------ [1] #1231 (comment) [2] https://github.com/notifications/unsubscribe-auth/AACFGMGZMM3UPQBRZI5Z7RLWZHHDVANCNFSM5KJWRTXQ

[mlangelaar]

Adding the PYTHONVERSION actually messes up the compile believe it or not, so here's an idea now, is it possible the PyInit_GDL is a false positive ? Where does it test for this ? Maiko

…

On 2023-02-25 03:36, Sylwester Arabas wrote: Googling the "dynamic module does not define module export function" message, it seems that in some cases this is related with Python version mismatch (the one that GDL module was compiled against vs. the one that is used when importing). In principle, there are three Python versions that must match: * the one CMake finds with find_package(PythonInterp) * the one CMake finds with find_package(PythonLibs) * the one used at runtime In the Debian Python module package source ("rules" file), they specify the following CMake options for building the module: -DPYTHON=YES -DPYTHONVERSION=3 -DPYTHON_MODULE=YES perhaps you could try with setting PYTHONVERSION=3.8.6 to make sure it is consistent? -- Reply to this email directly, view it on GitHub [1], or unsubscribe [2]. You are receiving this because you commented.Message ID: ***@***.***> Links: ------ [1] #1231 (comment) [2] https://github.com/notifications/unsubscribe-auth/AACFGMGZMM3UPQBRZI5Z7RLWZHHDVANCNFSM5KJWRTXQ

[slayoo]

Please provide logs from the failing compilation.
I think the PyInit_GDL is called (and the error message issued) around here: https://github.com/python/cpython/blob/main/Python/importdl.c#L150-L162

[mlangelaar]

Server freshly rebooted, yum updated, latest Centos 7.9, Python 3.8.13, gcc version 9.3.1 20200408 :

mkdir build ; cd build
cmake3 -DHDF5DIR=/usr/local/hdf5 -DQHULL=OFF -DPYTHON=YES -DPYTHON_MODULE=ON ..
make -j
make install

Then try and run it as follows :

[root@ssresearch build]# python -V
Python 3.8.13

[root@ssresearch build]# python
Python 3.8.13 (default, Aug 16 2022, 12:16:29)
[GCC 9.3.1 20200408 (Red Hat 9.3.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('/usr/local/lib/python3.8/site-packages')
>>> import GDL
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dynamic module does not define module export function (PyInit_GDL)
>>>

I have put the entire build directory on our webserver, which will include :

cmake.log, make.log, install.log, install manifest, etc ...

URL is http://www.physics.umanitoba.ca/build_ssresearch_gdl03Mar2023_maiko.tar.gz

IF you prefer I don't post a URL, please let me know ASAP and I will edit the post.

Maiko (no rush on this, I appreciate whatever you can do)

[slayoo]

In the logs you've pointed to, it is indicated that GDL compiles against Python 2.7 (cmake.log):

-- Found Python: /usr/bin/python2.7 (found version "2.7.5") found components: Interpreter Development NumPy 
...
-- Found PythonInterp: /opt/rh/rh-python38/root/usr/bin/python (found version "3.8.13") 
...
Python                 ON (libs:/usr/lib64/python2.7/config/libpython2.7.so; headers:/usr/include/python2.7)

The libraries pointed to must be consistent with the version of Python you use to import the module (i.e., 3.8.13 in your case).
Please point CMake to the correct version of Python, e.g., by using the PYTHONVERSION option to CMake.
Python needs to be installed with development files (headers, etc), for Fedora this is the python-devel package

[mlangelaar]

Change of strategy. the devtoolset 'suite' is causing me all the problems here, ironically the reason I went to it was
because I was getting compile errors using the OS stock gcc version 4.8.5 20150623 (Red Hat 4.8.5-44).

But quite honestly I decided just now to give up on the devtoolset, it's not working as far as development header
and libs, and I've wasted enough time trying to make it work with Python LIBRARIES and INCLUDE_DIRS and all
that other stuff.

So back to the OS stock python 3.6.8m (that's all I have to work with), the cmake is nice and clean now, and yes
the proper python and python3 devel packages are installed (for SL 7.9) so now I guess I just need to deal with
the make (compile errors) that made me use the higher version gcc in the devtoolset packages.

So here is the first one then it stops :

[ 54%] Building CXX object src/CMakeFiles/gdl.dir/histogram.cpp.o
/root/gdl/gdl/src/histogram.cpp: In instantiation of â?:
/root/gdl/gdl/src/histogram.cpp:313:56: required from here
/root/gdl/gdl/src/histogram.cpp:100:22: error: call of overloaded â? is ambiguous
if (isinf(val)) e->Throw("Illegal binsize or max/min.");
^
/root/gdl/gdl/src/histogram.cpp:100:22: note: candidates are:
In file included from /usr/include/features.h:375:0,
from /usr/include/c++/4.8.2/x86_64-redhat-linux/bits/os_defines.h:39,
from /usr/include/c++/4.8.2/x86_64-redhat-linux/bits/c++config.h:2097,
from /usr/include/c++/4.8.2/cstddef:41,
from /root/gdl/gdl/src/includefirst.hpp:45,
from /root/gdl/gdl/src/histogram.cpp:18:
/usr/include/bits/mathcalls.h:202:1: note: int isinf(double)
__MATHDECL_1 (int,isinf,, (Mdouble __value)) attribute ((const));
^
In file included from /usr/include/c++/4.8.2/random:38:0,
from /usr/include/c++/4.8.2/bits/stl_algo.h:65,
from /usr/include/c++/4.8.2/algorithm:62,
from /root/gdl/gdl/src/includefirst.hpp:47,
from /root/gdl/gdl/src/histogram.cpp:18:
/usr/include/c++/4.8.2/cmath:608:3: note: constexpr bool std::isinf(long double)
isinf(long double __x)
^
/usr/include/c++/4.8.2/cmath:604:3: note: constexpr bool std::isinf(double)
isinf(double __x)
^
/usr/include/c++/4.8.2/cmath:600:3: note: constexpr bool std::isinf(float)
isinf(float __x)
^
make[2]: *** [src/CMakeFiles/gdl.dir/histogram.cpp.o] Error 1
make[1]: *** [src/CMakeFiles/gdl.dir/all] Error 2
make: *** [all] Error 2

Sorry for the long delays between posts, busy guy, got way tooo much on my plate :(

Maiko

[slayoo]

@mlangelaar the error you reported:

error: call of overloaded â? is ambiguous
if (isinf(val)) ...

seems to be an old bug in GCC:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60407

Could you try with GCC >= 6 (i.e.,. a post-2015 release)

[mlangelaar]

that's why I chose to use devtoolset, which gave me gcc 9 I believe, I will have to
find a way to get the 'proper' devel files into place, that is really the issue here :(
There were a couple more errors, which I dealt with myself, but anyways ...
I'll be in touch, thank you.

[mlangelaar]

I have compiled a new gcc 9.2.0 from source and using it now. The next error I ran into was this one :

[ 15%] Building CXX object src/CMakeFiles/gdl.dir/hdf5_fun.cpp.o
/root/gdl/gdl/src/hdf5_fun.cpp: In destructor â?lib::hdf5_name_guard::~hdf5_name_guard()â?:
/root/gdl/gdl/src/hdf5_fun.cpp:113:26: error: â?H5free_memoryâ? was not declared in this scope
113 | ~hdf5_name_guard() { H5free_memory(name); }
| ^~~~~~~~~~~~~
make[2]: *** [src/CMakeFiles/gdl.dir/hdf5_fun.cpp.o] Error 1
make[1]: *** [src/CMakeFiles/gdl.dir/all] Error 2

I think we're getting there, because an earlier attempt gave me a GDL.so that actually got
past the Py_initGL() issue, so I think now I just need to get past this particular error.

Maiko

[slayoo]

Great! :)
The H5free_memory is part of the HDF5 API since version 1.8.13 (https://docs.hdfgroup.org/hdf5/v1_12/group___h5.html#title7).
(as reported here: #1517)
So, a newer version of HDF5 is needed or you can disable HDF5 support with CMake.
HTH

[mlangelaar]

I am using 1.8.13, but I had forgot to remove the devel packages for the previous version.

Anyways, I think I am past the py_initGL () part, which this discussion was originally about.

This is my latest results, and I think perhaps we can close this ? Unless this is happening
before it tries to do the py_initGL () call, which I don't think is the case, so yeah :

export CC=/opt/bin/gcc
export CXX=/opt/bin/g++

cmake3 -DQHULL=OFF -DHDF5=OFF -DNETCDF=OFF -DPYTHON=YES
-DPYTHONVERSION="3.6.8" -DPYTHON_MODULE=ON .. >& cmake.log

make -j ; make install

export LD_LIBRARY_PATH=/opt/lib64:/opt/lib (could be wrong)

[root@ssresearch build]# python3
Python 3.6.8 (default, Nov 10 2020, 07:30:01)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.

import sys
sys.path.append('/usr/local/lib/python3.8/site-packages')
import GDL
Traceback (most recent call last):
File "", line 1, in
ImportError: /usr/local/lib/python3.6/site-packages/GDL.so: undefined symbol: _ZThn712_N14wxTextCtrlBase8overflowEi

I do see the GCC 4.8.5 in there (interesting), anyways ...

This is where I don't know what to do anymore, could be my library paths are wrong for
the newly compiled gcc 9.2.0, that's my problem, and time to get a more recent linux distro
going on another box perhaps. Thank you all for your patience and help.

Maiko Langelaar / IT Physics and Astronomy (University of Manitoba)

• http://www.langelaar.net

[slayoo]

GCC 4.8.5 is the compiler with which the Python interpreter was compiled.
The missing wxTextCtrlBase8overflow symbol is related with widgets support in GDL which anyhow will likely not be adequate for using from within Python, so please try disabling it with a CMake flag (i.e., -DWXWIDGETS=OFF).

[mlangelaar]

That's what I kinda thought, but I was not sure.

1. Okay, so trying with -DWXWIDGETS=OFF, which gets me here :

   ImportError: /usr/local/lib/python3.6/site-packages/GDL.so: undefined symbol:
   _ZN6Magick5Image4readERKNSt7__cxx1112basic_stringIcSt11char_traits
   IcESaIcEEE

2. So let's try to add -DGRAPHICSMAGICK=OFF and -DMAGICK=OFF to the equation :

   [root@ssresearch build]# python3
   Python 3.6.8 (default, Nov 10 2020, 07:30:01)
   [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
   Type "help", "copyright", "credits" or "license" for more information.

   import sys
   sys.path.append('/usr/local/lib/python3.8/site-packages')
   import GDL
   a= GDL.function("sin",(1,))
   print(a)
   0.8414709568023682

3. They are only using the BINNING functions of GDL, so no graphics are needed (to my knowledge)

I owe you a coffee ! Thanks for your patience Sylwester.

(PS : my youngest kid was actually in Tomaszów Mazowiecki early February, Speed Skating ISU World Cup)

Maiko

[slayoo]

Great! Thanks for reporting all the issues - hope users can find answers here if someone bumps into similar problems.

(PS : my youngest kid was actually in Tomaszów Mazowiecki early February, Speed Skating ISU World Cup)

🇵🇱 :)

[mlangelaar]

just an UPDATE (no need to reply) - Now using Rocky 8.9

This compiles smoothly out of the box, and with epel-release, works with Python 3.9.13, no effort required :

cmake3 -DWXWIDGETS=OFF -DQHULL=OFF -DNETCDF=OFF -DGRAPHICSMAGICK=OFF \
  -DMAGICK=OFF -DPYTHON=YES -DPYTHON_MODULE=ON ..

 [root@nuit ~]# python3
 Python 3.9.13 (main, Nov 16 2022, 15:31:39)
 [GCC 8.5.0 20210514 (Red Hat 8.5.0-15)] on linux
 Type "help", "copyright", "credits" or "license" for more information.
 >>> import GDL
 >>> a= GDL.function("sin",(1,))
 >>> print(a)
 0.8414709568023682
 >>> quit()
 [root@nuit ~]#

None of this silly devtoolset, it's all stock packages using yum install, very nice.

Only one error (IDL actually) the user has run into, so if anyone knows this, although
seeing already he makes reference to a path IDL, so perhaps he forgot to rename an
IDL command ? No idea, I've never used this stuff before, I just compile and install.
(not related at all to this thread, but just incase someone is searching for it)

% Compiled module: GETTOK.
% Compiled module: VALID_NUM.
% Compiled module: MRD_SKIP.
% Compiled module: MATCH.
% MRD_DOFN: Function not found: IDL_VALIDNAME
% Error occurred at: MRD_DOFN 646 /home/xxxxx/IDL/IDLAstro-pro/mrdfits.pro
% MRD_TABLE 2369 /home/xxxxx/IDL/IDLAstro-pro/mrdfits.pro
% MRDFITS 2729 /home/xxxxx/IDL/IDLAstro-pro/mrdfits.pro
% GROUPPHAMOS 127 /disks/d2/xxxxxx/Test_Files/groupphamos.pro
% $MAIN$

Maiko

[GillesDuvert]

indeed, IDL_VALIDNAME is a procedure in GDL (not in IDL), so the !PATH must contain /usr/local/share/gnudatalanguage/lib (or anywhere GDL is installed)

[mlangelaar]

Success, the user reports :
"the path in questions was GDL_PATH, and just needed to be appended with the directory you gave"
So that's it folks - the user is able to run their script from start to end, flawless.
Thanks again !