Dialect: Add support for asyncpg and psycopg3 drivers
#11
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
amotl
force-pushed
the
amo/postgresql-async
branch
2 times, most recently
from
20ab16e
to
e1929cb
Compare
amotl
commented
Dec 23, 2023
tests/engine_test.py
Outdated
Comment on lines
24
to
43
| def test_engine_sync_vanilla(): | ||
| """ | ||
| crate:// -- Verify connectivity and data transport with vanilla HTTP-based driver. | ||
| """ | ||
| engine = sa.create_engine("crate://crate@localhost:4200/", echo=True) | ||
| assert isinstance(engine, sa.engine.Engine) | ||
| with engine.connect() as connection: | ||
| result = connection.execute(QUERY) | ||
| assert result.mappings().fetchone() == {'mountain': 'Acherkogel', 'coordinates': [10.95667, 47.18917]} | ||
|
|
||
|
|
||
| def test_engine_sync_urllib3(): | ||
| """ | ||
| crate+urllib3:// -- Verify connectivity and data transport *explicitly* selecting the HTTP driver. | ||
| """ | ||
| engine = sa.create_engine("crate+urllib3://crate@localhost:4200/", isolation_level="AUTOCOMMIT", echo=True) | ||
| assert isinstance(engine, sa.engine.Engine) | ||
| with engine.connect() as connection: | ||
| result = connection.execute(QUERY) | ||
| assert result.mappings().fetchone() == {'mountain': 'Acherkogel', 'coordinates': [10.95667, 47.18917]} |
There was a problem hiding this comment.
This is the reference record representation.
{'mountain': 'Acherkogel', 'coordinates': [10.95667, 47.18917]}
tests/engine_test.py
Outdated
Comment on lines
46
to
67
| def test_engine_sync_psycopg(): | ||
| """ | ||
| crate+psycopg:// -- Verify connectivity and data transport using the psycopg driver (version 3). | ||
| """ | ||
| engine = sa.create_engine("crate+psycopg://crate@localhost:5432/", isolation_level="AUTOCOMMIT", echo=True) | ||
| assert isinstance(engine, sa.engine.Engine) | ||
| with engine.connect() as connection: | ||
| result = connection.execute(QUERY) | ||
| assert result.mappings().fetchone() == {'mountain': 'Acherkogel', 'coordinates': '(10.95667,47.18917)'} | ||
|
|
||
|
|
||
| @pytest.mark.asyncio | ||
| async def test_engine_async_psycopg(): | ||
| """ | ||
| crate+psycopg:// -- Verify connectivity and data transport using the psycopg driver (version 3). | ||
| This time, in asynchronous mode. | ||
| """ | ||
| engine = create_async_engine("crate+psycopg://crate@localhost:5432/", isolation_level="AUTOCOMMIT", echo=True) | ||
| assert isinstance(engine, AsyncEngine) | ||
| async with engine.begin() as conn: | ||
| result = await conn.execute(QUERY) | ||
| assert result.mappings().fetchone() == {'mountain': 'Acherkogel', 'coordinates': '(10.95667,47.18917)'} |
There was a problem hiding this comment.
When using psycopg, there are deviations, which clearly need to be addressed. By chance, you might have seen this already, @SStorm? Or did you only use asyncpg?
{'mountain': 'Acherkogel', 'coordinates': '(10.95667,47.18917)'}
tests/engine_test.py
Outdated
Comment on lines
70
to
81
| @pytest.mark.asyncio | ||
| async def test_engine_async_asyncpg(): | ||
| """ | ||
| crate+asyncpg:// -- Verify connectivity and data transport using the asyncpg driver. | ||
| This exclusively uses asynchronous mode. | ||
| """ | ||
| from asyncpg.pgproto.types import Point | ||
| engine = create_async_engine("crate+asyncpg://crate@localhost:5432/", isolation_level="AUTOCOMMIT", echo=True) | ||
| assert isinstance(engine, AsyncEngine) | ||
| async with engine.begin() as conn: | ||
| result = await conn.execute(QUERY) | ||
| assert result.mappings().fetchone() == {'mountain': 'Acherkogel', 'coordinates': Point(10.95667, 47.18917)} |
There was a problem hiding this comment.
When using asyncpg, the response record is not wrong, but its shape is different, as it provides a dedicated Point type for representing coordinates.
{'mountain': 'Acherkogel', 'coordinates': Point(10.95667, 47.18917)}
This was referenced Dec 23, 2023
amotl
changed the title
psycopg and asyncpg driversasyncpg and psycopg drivers
amotl
changed the title
asyncpg and psycopg driversasyncpg and psycopg3 drivers
amotl
force-pushed
the
amo/fix-inspector
branch
from
2f08b2a
to
43c45fb
Compare
amotl
changed the title
asyncpg and psycopg3 driversasyncpg and psycopg3 drivers
amotl
force-pushed
the
amo/fix-inspector
branch
5 times, most recently
from
6ac0a22
to
d2c7613
Compare
amotl
force-pushed
the
amo/postgresql-async
branch
3 times, most recently
from
5c906ad
to
474f658
Compare
amotl
force-pushed
the
amo/postgresql-async
branch
from
474f658
to
dcb6708
Compare
proddata
reviewed
Jun 25, 2024
Comment on lines
+403
to
+412
| When using the PostgreSQL protocol with drivers `psycopg` or `asyncpg`, | ||
| the paramstyle is not `qmark`, but `pyformat`. |
There was a problem hiding this comment.
I am not quite following. I assume we would want to use server side binding (i.e. qmark)?
There was a problem hiding this comment.
This is probably the reason for needing the workaround at all: Because PostgreSQL drivers psycopg and asyncpg, or the SA dialect, use pyformat, but CrateDB uses qmark, we may need to adjust, iirc.
At least, the patch in its current shape needs it. Maybe there are alternatives to implement it, possibly even easier ones. We will be happy to learn about them.
amotl
force-pushed
the
amo/postgresql-async
branch
from
dcb6708
to
e8bfd77
Compare
amotl
force-pushed
the
amo/postgresql-async
branch
from
e8bfd77
to
f61ebaa
Compare
This introduces the `crate+psycopg://`, `crate+asyncpg://`, and `crate+urllib3://` dialect identifiers. The asynchronous variant of `psycopg` is also supported.
amotl
force-pushed
the
amo/postgresql-async
branch
from
f61ebaa
to
bebbf07
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
About
This patch adds support for asyncpg and psycopg3 drivers, by introducing the
crate+asyncpg://andcrate+psycopg://dialect identifiers.It also adds the
crate+urllib3://dialect identifier, for explicitly addressing the urllib3-based transport, in case there might be other HTTP-based transport adapters in the future, using libraries like aiohttp, httpx, or niquests. 1Notes
The asynchronous variant of
psycopgis also supported and will be automatically selected when usingcreate_async_engine()instead ofcreate_engine(), so it doesn't have a dedicated dialect identifier.All of this will only work with SQLAlchemy 2.x.
Installation
pip install 'sqlalchemy-cratedb[all] @ git+https://github.com/crate/sqlalchemy-cratedb@amo/postgresql-async'References
Backlog
psycopg, see below.asyncpgandpsycopg3cratedb-examples#201pandas.read_sql()with both urllib3 vs. psycopg3 cratedb-examples#651Footnotes
Picked up from another discussion at https://github.com/panodata/grafana-client/issues/134. ↩