Pyarrow directly

[WillAyd]

closes #xxx

[WillAyd]

@jorwoods we talked about this previously would love your input.

This makes it so you can write pyarrow tables, and in theory anything that implements the PyArrow Capsule Interface

The writing is easy - we can just check for the __arrow_c_stream__ dunder on an object and work off of that (pandas will have that in their 3.0 release, polars has an open request for it

I am less sure about what to do when reading the resuilt. Right now this still converts everything back to a pandas dataframe, simply owing to the history of this library. Other viable alternatives are:

• Return a PyArrow table; users can convert to whatever from there
• Accept a keyword argument like return_type = "pandas|pyarrow|polars"

The return_type argument is convenient but maybe too presumptive about how those libraries handle conversions. The base pyarrow type is the safest but maybe the least convenient for end users.

Curious what you might have in mind

[jorwoods]

I think returning the pyarrow table is the least presumptive about what the end user needs. This would allow a little more flexibility in the users grabbing what data type they need. Could add some basic convenience functions for quickly reading it into pandas though.