Add software tests using Polars #35
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #35 +/- ##
=======================================
Coverage 97.56% 97.56%
=======================================
Files 3 3
Lines 41 41
=======================================
Hits 40 40
Misses 1 1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
| def test_psycopg_read_sql(cratedb_psql_host, cratedb_psql_port): | ||
| engine = sa.create_engine( | ||
| url=f"postgresql+psycopg_relaxed://crate@{cratedb_psql_host}:{cratedb_psql_port}", | ||
| isolation_level="AUTOCOMMIT", | ||
| use_native_hstore=False, | ||
| echo=True, | ||
| ) | ||
| conn = engine.connect() | ||
| df = pl.read_database( | ||
| query=SQL_SELECT_STATEMENT, | ||
| connection=conn, | ||
| schema_overrides={"value": pl.Utf8}, | ||
| ) | ||
| assert_frame_equal(df, REFERENCE_FRAME) |
There was a problem hiding this comment.
Dear @surister,
I am only now reading your comment over at crate/crate#12952 (comment). Can I humbly ask you to check whether the operation on 600K rows worth of data is faster using the Psycopg3 driver, compared to the HTTP/JSON-based driver from crate-python?
pip install 'polars[pyarrow]<0.21' sqlalchemy-postgresql-relaxedimport polars as pl
import sqlalchemy as sa
engine = sa.create_engine("postgresql+psycopg_relaxed://crate@localhost/doc")
df = pl.read_database(
query="SELECT * FROM metrics",
connection=engine,
schema_overrides={"value": pl.Utf8},
)
print(df)With kind regards,
Andreas.
NB: This will not run the CrateDB SQLAlchemy dialect though, so it will probably not support any special data types yet. There is another patch which will bring everything together correspondingly.
This patch will, for example, unlock connectivity using the CrateDB SQLAlchemy dialect together with the Psycopg3 database driver.
engine = sa.create_engine("crate+psycopg://crate@localhost/doc")
There was a problem hiding this comment.
- read_database - postgresql+psycopg_relaxed 8.5s
- read_database_uri - postgresql 5.13s
- read_database - crate 8.8s
Polars does not support writing to databases yet, nor does it support SQLAlchemy's AsyncEngine. So, this exercises `pl.read_database()` only, together with the `crate` and `postgresql+psycopg_relaxed` dialects, using urllib3 resp. Psycopg3.
amotl
force-pushed
the
amo/test-polars
branch
from
973cd13
to
93b2a8d
Compare
About
Following up on validating pandas, this patch starts validating the Polars DataFrame library.
Details
Polars does not support writing to databases yet, nor does it support SQLAlchemy's AsyncEngine. So, this exercises
pl.read_database()only, together with thecrateandpostgresql+psycopg_relaxeddialects, using urllib3 resp. Psycopg3.References
asyncpgandpsycopg3drivers crate/sqlalchemy-cratedb#11/cc @surister, @SStorm