Upcoming breaking changes in Numpy 2.0

[corykinney]

NumPy 2.0 hasn't been released yet, but the initial release notes from the pre-release mentions some breaking changes that they address in their migration guide. I don't know how the Cantera team wants to handle compatibility going forward, but I thought I would go ahead and make a thread in case it's helpful.

Previous Cantera builds

I installed NumPy 2.0.0b1 and attempted to import Cantera 3.0.0 which yields the following error:

...
    from ._cantera import *
  File "cantera/_cantera.pyx", line 33, in init cantera._cantera
  File "cantera/_utils.pyx", line 1, in init cantera._utils
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

Clearly previous builds of Cantera can only be used with NumPy 1.x and will not work with 2.x releases. I don't know if there is a way to enforce numpy<2.0.0 as a package requirement on PyPI, or if that can't be done retroactively, but it might need to be added to some documentation somewhere.

Future Cantera builds

I attempted to build Cantera from source from the latest commit using NumPy 2.0.0b1 to see what changes would need to be made. The first build error that I get from the breaking changes is below:

build\python\cantera\_onedim.cpp(8544): error C2039: 'subarray': is not a member of '_PyArray_Descr'
...
build\python\cantera\_onedim.cpp(8545): error C2039: 'subarray': is not a member of '_PyArray_Descr'

The changes to the PyArray_Descr struct are covered in this section of the migration guide. It seems the PyDataType_SUBARRAY accessor function needs to be used. They also provide a npy2_compat.h file that can be included in the code base that will allow compilation on NumPy 1.x and not affect 2.x builds.

I haven't gotten any further yet, but I will edit this if I do.

[speth]

Clearly previous builds of Cantera can only be used with NumPy 1.x and will not work with 2.x releases. I don't know if there is a way to enforce numpy<2.0.0 as a package requirement on PyPI, or if that can't be done retroactively, but it might need to be added to some documentation somewhere.

We would need to rebuild the PyPI packages to add that version restriction. How many things are broken in that workflow is an interesting question.

It seems the PyDataType_SUBARRAY accessor function needs to be used. They also provide a npy2_compat.h file that can be included in the code base that will allow compilation on NumPy 1.x and not affect 2.x builds.

Those errors are coming from Cython-generated code, so I think we just need to wait for Cython to be updated on this one. What version of Cython are you using?

[corykinney]

Those errors are coming from Cython-generated code, so I think we just need to wait for Cython to be updated on this one. What version of Cython are you using?

Sorry, I'm still getting used to Cython, so I did not realize that the error was coming from that instead of the onedim Cantera file, which explains why I couldn't find it.
I am using Cython 3.0.9, which I thought should work based on this from the migration guide:

Where not possible, new accessor functions are required:
...

• PyDataType_FIELDS, PyDataType_NAMES, PyDataType_SUBARRAY

...
Cython code should use Cython 3, in which case the change is transparent.

Perhaps I should open an issue on Cython instead?

[ischoegl]

To add my 2 cents, I don't think that we need to jump on implementing changes based on a beta release yet. At the same time, we have an established workflow for prerelease dependencies in our post-merge-tests.yml CI tests. While it is released now, Cython 3.0 was in pre-release until recently, and we had a test ensuring that things would compile (see prerelease-cython runner); while this used to be part of the main CI job, we moved it to a post-merge test as it was considered non-critical.

I would suggest to simply modify the (now no longer needed) CI runner for pre-release Cython to test for NumPy 2.x instead, so we can monitor whether things clear up (breaking NumPy API changes may or may not be breaking for Cantera due to the Cython layer, which at this point should default to 3.0.x). This modification is presumably quite simple - a PR is welcome and a good intro to learn about our GH actions tests.

Beyond, NumPy v2.0.0b1 was just released 2 days ago. While I found this statement "The tentative release date for the first release candidate of NumPy 2.0 is around 1 Mar 2024, and the final release 6-8 weeks later," it is not necessarily easy to predict how fast downstream packages are expected/able to move. I consider Cantera as downstream relative to Cython and other packages that themselves depend on NumPy, so we may have to pin to numpy<2.0 until things stabilize upstream. As an aside, I personally do not believe that we need to make the upcoming Cantera 3.1 release contingent on support for NumPy 2.0.

[bryanwweber]

My 2 cents as well... 😄 If this dependency cap is added to 3.0 (by creating .post1, most likely) this may also be a good time to add Python 3.12 support to the builds, and GitHub Actions supports native ARM Mac runners now, which should simplify cross-compiling. See Cantera/enhancements#198 I suppose it depends on the timeline for 3.1.

[ischoegl]

My 2 cents as well... 😄 If this dependency cap is added to 3.0 (by creating .post1, most likely) this may also be a good time to add Python 3.12 support to the builds, and GitHub Actions supports native ARM Mac runners now, which should simplify cross-compiling. See Cantera/enhancements#198 I suppose it depends on the timeline for 3.1.

Great to hear that you're volunteering for .post1! 😜

Jests aside the fact that we pretty much decided to remove the legacy MATLAB interface (#1670) may mean that 3.0 may be around for a while for some users. At the same time I am very hesitant to spend time on back ports.

[bryanwweber]

Great to hear that you're volunteering for .post1

Happy to help as much as I can with this. I know the packaging stuff is still pretty arcane. ❤️

[bryanwweber]

As a point of reference, conda-forge attempted an automatic upgrade to NumPy 2.0 and as expected it failed. conda-forge/cantera-feedstock#38

[bryanwweber]

With conda-forge/cantera-feedstock#38 and #1719 merged, there should be packages published for 3.0.1 on PyPI that back port changes to support NumPy 2.0 and Python 3.12 and 3.13. The last step is to update the Cantera channel conda recipes to build 3.0.1 and pin NumPy.

[bryanwweber]

Alright, Cantera/conda-recipes#50 is merged which pins NumPy < 2.0 for our conda packages. Since the defaults channel is behind and doesn't have NumPy 2.0 yet, we're deferring that work (possibly forever since conda-forge is well supported by now 😄).

This work is done, thanks for raising it!