Current clifford master does not pass unit tests with Numba 0.52.0rc2

[esc]

It always seems to be same issue, something related to Numba and how functions are pickled. The log is attached below. I am trying to figure out if this is an issue with the Numba 0.52.0rc2 or with Clifford.

Reproducer

# Assuming we are in a git clone of clifford, have conda and all dependencies installed.
$ git desc
v1.3.0-67-g22192841f9

$ conda list | grep "numba\|clifford"
# packages in environment at /Users/esc/miniconda3/envs/clifford:
clifford                  1.4.0.dev0               pypi_0    pypi
llvmlite                  0.35.0rc2                py37_0    numba/label/dev
numba                     0.52.0rc2       np1.11py3.7h6440ff4_gd5984fa_0    numba/label/dev

$ pytest -v clifford/test/test_tools_classify.py::TestCGA3D
LOG ATTACHED BELOW

LOG.txt

[eric-wieser]

Reading the log, the error is the following repeated a lot

_pickle.PicklingError: Recursive function reference on <function Layout._grade_invol.<locals>.grade_inv_func at 0x7f8baf3e2dd0>

Interestingly, the source of that error *numba code) contains the comment:

            # Detect recursive pickling; i.e. function references itself.
            # NOTE: this is not ideal, but we prefer the fast pickler which
            #       does this properly.

Why doesn't the slow pickler do this properly too? Shouldn't the slow one be a superset of the fast one?

[sklam]

The slow pickler is actually referring to pickle._Pickler which is the pure-python Pickler but is slightly more extensible before Py3.8 than the fast pickler (pickle.Pickler; the default pickler). In Numba, we use the slow pickler in Py<=3.7 unless there is pickle5 in the environment. The slow pickler just does not have enough facility and documentation to allow us to handle the recursion function case for a lot of the situation. That is why we prefer the fast pickler.

I don't think there were any changes in numba 0.52 that is related to the pickler in numba. @esc, Did clifford work fine in 0.51?

[esc]

@sklam, i just checked and the following config fails these tests too:

 $ conda list | grep "numba\|clifford\|llvmlite"
# packages in environment at /Users/esc/miniconda3/envs/clifford:
clifford                  1.4.0.dev0               pypi_0    pypi
llvmlite                  0.34.0           py37h739e7dc_4    defaults
numba                     0.51.2           py37h959d312_1    defaults

[esc]

@sklam is there anything that can be done to remedy this on the Numba side?

[sklam]

I don't think we can fix it that quickly. We should open an issue about it.