Allow sinking and scanning of lazyframes

[skyth540]

Is your feature request related to a problem? Please describe.
Lazyframes allow for larger-than-memory dataframes to be handled. pantab does not support lazyframes

Describe the solution you'd like
a sink_to_hyper function similar to sink_parquet and a scan_from_hyper function similar to scan_parquet would be very useful for projects involving lazyframes.

Describe alternatives you've considered
I've tried to sink_parquet and use this library to convert, however it doesn't work with my file, and that takes extra time and overhead anyhow

[WillAyd]

In pantab we don't necessarily want to tie ourselves to any of the libraries this strongly; we just provide one interface and it is up to the library to implement the Arrow C Data specification

I think this is more of a question for polars. You may want to ask about this upstream in an issue like pola-rs/polars#12530 and see what plans they have for implementing that for Lazyframes

[WillAyd]

Closing as I don't think there is anything we can do about this in pantab. For libraries to be usable in pantab, they must conform to the Arrow PyCapsule Interface:

https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html

So if desired I think should ask for that adherence with LazyFrames upstream

[skyth540]

Sounds good. I broke my project into chunks and iterated myself

…

On Thu, Oct 10, 2024, 11:56 AM William Ayd ***@***.***> wrote: Closing as I don't think there is anything we can do about this in pantab. For libraries to be usable in pandas, they must conform to the Arrow PyCapsule Interface: https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html So if desired I think should ask for that adherence with LazyFrames upstream — Reply to this email directly, view it on GitHub <#346 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXRFQKVSAXUJ6MBVEJ2OVC3Z225WNAVCNFSM6AAAAABO7F3EZOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBVG4ZDGOJVGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>