docs(python): Expand NumPy section in the user guide with ufunc info

[deanm0000]

scipy.special is (almost) all ufuncs so I provided a link there.

numba creates ufuncs so I provided a link to its page and an example thereof.

Lastly, added a note about map_batches avoidance. For the longest time I thought avoiding map_batches was a way to improve performance until I stumbled on to this which shows it just calls map_batches anyway.

[MarcoGorelli]

so glad you've opened a PR about this 🙌

I think the lint failure is just because you have a trailing space at the end of this line

[deanm0000]

I guess I'll fix this when numba can support the latest python version as that failed too.

[deanm0000]

looks like numba doesn't work on python 3.12 yet so my numba example won't work unless the example can be run on a lower version of python.

[MarcoGorelli]

looks like it's coming soon - tbh I'd suggest just waiting numba/numba#9197

[deanm0000]

Yeah I saw that and back burnered it unless/until something happens

[stinodego]

For examples that can't run during docs building (e.g. downloading stuff from the Cloud), you can enclose them in triple quotes. See this file for an example:
https://github.com/pola-rs/polars/blob/main/docs/src/python/user-guide/lazy/execution.py

[henryharbeck]

Hi @deanm0000 and @MarcoGorelli
Numba now supports python 3.12 since their 0.59.0 release.
Release notes are here.
It seems like there are still a few post-relase tasks to go, such as creating a github release (ref: numba/numba#9410), but I believe that this PR would now be unblocked.

[deanm0000]

Hey thanks for the heads up. I'm working on another ufunc change so I'll probably sit on this unless/until that gets merged so that I don't duplicate too much work.

Here's the aforementioned PR #14328

[MarcoGorelli]

I'm confused by this example - for these particular numbers, they're all above the threshold, so wouldn't the result just be the same as the input? maybe run it with cum_sum_reset(pl.all(), 30)?

[MarcoGorelli]

true, but map_batches has is_elementwise=False by default, and so will do the expected thing in group-by / over

maybe, rather than avoiding map_batches, map_batches should be the primary way the ufuncs are taught?

[deanm0000]

@MarcoGorelli

It turns out that all the numpy and scipy ufuncs are element wise in the sense that they aren't aggregations. Where this becomes important is if someone wants to do a sum and then the np.expm1 function. The ufuncarray hook will do is_elementwise=true. If it's anumba ufunc then it won't.

Whether people should be taught to not use the hook is more philosophical imo. I suppose there's slightly less parsing in that case so would be technically more performant but by using it, it's a nice syntax, imo.

[MarcoGorelli]

this isn't true anymore, is it?

In [18]: df.with_columns(result=pl.col('b').map_batches(lambda x: np.cumsum(x)).over('a'))
Out[18]:
shape: (3, 3)
┌─────┬─────┬────────┐
│ a   ┆ b   ┆ result │
│ --- ┆ --- ┆ ---    │
│ i64 ┆ i64 ┆ i64    │
╞═════╪═════╪════════╡
│ 1   ┆ 4   ┆ 4      │
│ 1   ┆ 5   ┆ 9      │
│ 2   ┆ 6   ┆ 6      │
└─────┴─────┴────────┘

[deanm0000]

You're right. I've lost quite a bit of steam on this.

[MarcoGorelli]

Hey

Sorry, looking at this again, and to be honest I'm not totally sold that numba should be advertised in the users guide until the null value handling is sorted out (#14811)

Otherwise results like this

import polars as pl
import numba as nb

df = pl.DataFrame({"a": [40, 39, None, 37]}) - 30


@nb.guvectorize([(nb.int64[:], nb.int64, nb.int64[:])], "(n),()->(n)")
def cum_sum_reset(x, y, res):
    res[0] = x[0]
    for i in range(1, x.shape[0]):
        res[i] = x[i] + res[i - 1]
        if res[i] >= y:
            res[i] = x[i]


out = df.with_columns(result=cum_sum_reset(pl.all(), 30))
print(out)

import polars as pl
import numba as nb

df = pl.DataFrame({"a": [40, 39, None, 37]}) - 30


@nb.guvectorize([(nb.int64[:], nb.int64, nb.int64[:])], "(n),()->(n)")
def cum_sum_reset(x, y, res):
    res[0] = x[0]
    for i in range(1, x.shape[0]):
        res[i] = x[i] + res[i - 1]
        if res[i] >= y:
            res[i] = x[i]


out = df.with_columns(result=cum_sum_reset(pl.all(), 30))
print(out)

look a bit scary

[itamarst]

#15194 is my attempt at dealing with this document.

[deanm0000]

I'll defer