Support for pl.List('*')

[bobir01]

Description

I noticed that in dataframe selection it is not possible to select columns by by_dtype (using selectors) when column is NESTED_DTYPES.
So why there is no such option of selecting by_dtype in higher levels like pl.List('*') or pl.List(pl.Any) (i know it does not exist)

I don't know much about Rust but even in python level, there is dummy solution :)

[(key, val) for key, val in df.schema.items() if val == pl.List]
############ OUTPUT #############
[('barcode_list', List(Binary)),
 ('front_category_list', List(Binary)),
 ('substitution_product_list', List(Binary)),
 ('image_list', List(Binary))]

I figured out that in order to select pl.List columns we should specify the inner dtypes like:
pl.List(pl.String)

could someone tell is_dtype selection will be in python level or rust level?

Thank you for your time!

[stinodego]

Passing pl.List to the by_dtype selector should work. I'll classify this as a bug. Thanks for the report!

The difficulty here is that, on the Rust side, we don't really have the notion of a generic uninstantiated List type.

[bobir01]

@stinodego , this issue is happening with datetime too ;(

[cmdlineluser]

For datetimes specifically, there is the dedicated .datetime() selector.

cs.by_dtype(pl.Datetime).meta.serialize()
# {'DtypeColumn':[{'Datetime':['Microseconds',null]}]}

cs.datetime().meta.serialize()
# {'DtypeColumn': [{'Datetime': ['Milliseconds', '*']},
#  {'Datetime': ['Milliseconds', None]},
#  {'Datetime': ['Microseconds', '*']},
#  {'Datetime': ['Microseconds', None]},
#  {'Datetime': ['Nanoseconds', '*']},
#  {'Datetime': ['Nanoseconds', None]}]}

It uses * to wildcard timezones, and None for the no timezone case.

It also has to do it for each possible time_unit.

The chat in #13683 may be relevant.

[bobir01]

For datetimes specifically, there is the dedicated .datetime() selector

@stinodego ohh i was using pl.Datetime therefore i might missed this Thank you i didnt now about cs.datetime()

[moritzwilksch]

Bumping since as of 1.2.1 polars raises a warning that the NESTED_TYPES constant is deprecated:

DeprecationWarning: `NESTED_DTYPES` is deprecated. Define your own data type groups or use the `polars.selectors` module for selecting columns of a certain data type

so dtype in pl.NESTED_DTYPES (which was a recommendation made in a previous warning iirc) is no longer supported #11844