Investigate nanobind in place of pybind11

[brendan-ward]

@benbovy feel free to close this if you are already aware of it and didn't want to consider it. I'm hoping to carve out a little time at some point to see what it might take to switch and whether or not it would be helpful; don't feel this issue is an obligation for you to do so.

nanobind is effectively a newer, smaller, lighter weight replacement of pybind11; the why page helps explain why it was created. The core API is very similar to pybind11 but at present is a bit less full-featured (which is partly a good thing). It targets a narrower range of C++ (17+) and Python (3.8+) versions. The potential downsides are that it is newer and less-full featured, and hasn't yet accumulated lots of GH issues / stack overflow posts around less common situations, compared to pybind11 (though these may not matter).

I'm not yet sure if the ndarray interface in nanobind would make some of the challenges around numpy and dtypes easier or harder.

I recently ported a small library from pybind11 to nanobind, and overall the experience was smooth and produced wheel files that were a little smaller (did not benchmark compile or runtime timings).

[benbovy]

@brendan-ward thanks for opening this issue. Nanobind has been briefly discussed in various places but it is better to gather the discussion here. Also good to know that the transition from pybind11 to nanobind went smoothly in your case.

A few thoughts:

• Smaller binaries would be nice
• I think one of the main benefits of using nanobind here would be its low-level interface. It would allow accessing the Python Geography types from C++ in a much cleaner way than the hacks used here to get the type information. This would help for efficient Python <-> C++ conversion of Geography objects (C++ <-> Python conversion performance #3).
• C++17 is fine I think, it is already used for other packages in the stack (s2geometry, s2geography, arrow, etc.) at least in conda-forge. Python 3.8+ is fine too for a new library like Spherely.
• I'm not sure either about nanobind support for numpy / dtypes, but one main limitation is that currently there's no equivalent of pybind11::vectorize, which makes things super easy here for creating numpy ufunc-like functions. There's a few options:
  • Wait for nanobind to provide a nb::vectorize helper, although I don't know if that's part of their roadmap.
  • Copy the pybind11::vectorize implementation in Spherely and adapt it to nanobind. This would probably help releasing the GIL in vectorized functions that create new Geography objects (pybind11:vectorize and GIL release #2). The pybind11::vectorize implementation is quite complex, though, this is far from ideal if we have to maintain an adapted version here (even if simplified).
  • Use xtensor's xt::vectorize along with xtensor-python. The latter has no support for nanobind yet, though (Add support for nanobind? xtensor-stack/xtensor-python#284).

[benbovy]

One thing that currently prevents us using nanobind here is that it doesn't supports arbitrary Python objects as numpy array items.

[benbovy]

I'm closing this issue since nanobind doesn't seem to have a plan for adding support of numpy arrays with arbitrary Python objects anytime soon. Let's re-open it if that changes.