Types: Unlock supporting timezone-aware `DateTime` fields #22

amotl · 2024-01-15T23:04:29Z

About

Timezone-aware SQLAlchemy DateTime fields apparently are supported already. This patch just removes a corresponding assertion which prevented them from actually being used.

References

Originally, this is coming from a monkeypatch to meltano-target-cratedb. Most probably, it is also related to those requests:

Other than this, that other patch is also related:

Types: Accept marshalling datetime.date values on DateTime fields #25

Backlog

Software tests.
Changelog item.
Check if documentation needs to be updated.

matriv

Left some comments, thx.

CHANGES.md

tests/datetime_test.py

matriv · 2024-06-24T07:10:50Z

tests/datetime_test.py

+    assert result["date"].year == 2009
+    assert result["datetime"].year == 2009
+    assert result["datetime"].tzname() is None
+    assert result["datetime"].timetz() == dt.time(19, 19, 30)


I don't see any conversion, using the CST timezone, the tz is not set on the result (as it should be) but the value stored shouldn't it be converted to UTC first ? (so hour should be 18?)

When inserting or updating records using timezone-aware Python date or datetime objects, timezone information will not be preserved.

-- https://cratedb.com/docs/sqlalchemy-cratedb/data-types.html

Shouldn't it be converted to UTC first?

I am not sure. We may want to validate how it works with PostgreSQL and its SQLAlchemy dialect?

We need to use timestamp WITH time zone as the column data type to make the conversion, otherwise the time zone information is just stripped:

cr> create table test(id string, tz timestamp with time zone); CREATE OK, 1 row affected (0.359 sec) cr> insert into test values('UTC1', '2024-06-24 10:20:30.123UTC'); INSERT OK, 1 row affected (0.017 sec) cr> insert into test values('UTC2', '2024-06-24 10:20:30.123'); INSERT OK, 1 row affected (0.017 sec) cr> insert into test values('+2h', '2024-06-24 10:20:30.123+02:00'); INSERT OK, 1 row affected (0.009 sec) cr> insert into test values('+3h', '2024-06-24 10:20:30.123+03:00'); INSERT OK, 1 row affected (0.018 sec) cr> select * from test order by id; +------+---------------+ | id | tz | +------+---------------+ | +2h | 1719217230123 | | +3h | 1719213630123 | | UTC1 | 1719224430123 | | UTC2 | 1719224430123 | +------+---------------+ SELECT 4 rows in set (0.004 sec) cr> drop table test; DROP OK, 1 row affected (0.045 sec) cr> create table test(id string, tz timestamp without time zone); CREATE OK, 1 row affected (0.361 sec) cr> insert into test values('UTC1', '2024-06-24 10:20:30.123UTC'); INSERT OK, 1 row affected (0.017 sec) cr> insert into test values('UTC2', '2024-06-24 10:20:30.123'); INSERT OK, 1 row affected (0.017 sec) cr> insert into test values('+2h', '2024-06-24 10:20:30.123+02:00'); INSERT OK, 1 row affected (0.018 sec) cr> insert into test values('+3h', '2024-06-24 10:20:30.123+03:00'); INSERT OK, 1 row affected (0.017 sec) cr> select * from test order by id; +------+---------------+ | id | tz | +------+---------------+ | +2h | 1719224430123 | | +3h | 1719224430123 | | UTC1 | 1719224430123 | | UTC2 | 1719224430123 | +------+---------------+ SELECT 4 rows in set (0.004 sec)

Can we have a test with time zone for the col type?

fyi, we follow the behavior of postgres:

matriv=> create table test(id text, tz timestamp without time zone); CREATE TABLE matriv=> insert into test values('UTC1', '2024-06-24 10:20:30.123UTC'); INSERT 0 1 matriv=> insert into test values('UTC2', '2024-06-24 10:20:30.123'); INSERT 0 1 matriv=> insert into test values('+2hr', '2024-06-24 10:20:30.123+02:00'); INSERT 0 1 matriv=> insert into test values('+3hr', '2024-06-24 10:20:30.123+03:00'); INSERT 0 1 matriv=> select * from test; id | tz ------+------------------------- UTC1 | 2024-06-24 10:20:30.123 UTC2 | 2024-06-24 10:20:30.123 +2hr | 2024-06-24 10:20:30.123 +3hr | 2024-06-24 10:20:30.123 (4 rows) matriv=> drop table test; DROP TABLE matriv=> create table test(id text, tz timestamp with time zone); CREATE TABLE matriv=> insert into test values('UTC1', '2024-06-24 10:20:30.123UTC'); INSERT 0 1 matriv=> insert into test values('UTC2', '2024-06-24 10:20:30.123'); INSERT 0 1 matriv=> insert into test values('+2hr', '2024-06-24 10:20:30.123+02:00'); INSERT 0 1 matriv=> insert into test values('+3hr', '2024-06-24 10:20:30.123+03:00'); INSERT 0 1 matriv=> select * from test; id | tz ------+---------------------------- UTC1 | 2024-06-24 13:20:30.123+03 UTC2 | 2024-06-24 10:20:30.123+03 +2hr | 2024-06-24 11:20:30.123+03 +3hr | 2024-06-24 10:20:30.123+03 (4 rows)

Thanks for your suggestions. I will add more test cases, and also validate the implementation using the whole set of possible datetime/timestamp column types.

619c991, added just now, starts using time zone Europe/Kyiv for testing, but it does not change anything on the test outcome.

I've inspected the wire using sudo ngrep -d lo0 -Wbyline port 4200, and found that on the JSON level, serialization does not take the time zone into account when invoking the software test test_datetime_tz using pytest --no-cov -k test_datetime_tz?

The library just sends 2009-05-13T19:19:30.123456Z to the wire, effectively ignoring time zone information at data submission time already. It is clear that there is a flaw.

{"stmt": "INSERT INTO foobar (name, date, datetime_notz, datetime_tz) VALUES (?, ?, ?, ?)", "args": ["foo", "2009-05-13", "2009-05-13T19:19:30.123456Z", "2009-05-13T19:19:30.123456Z"]}

Contrary to the observation above, like @matriv suggested, it looks like it works well when using direct SQL for querying.

DDL

SHOW CREATE TABLE doc.foobar;

CREATE TABLE IF NOT EXISTS "doc"."foobar" ( "name" TEXT NOT NULL, "date" TIMESTAMP WITHOUT TIME ZONE, "datetime_notz" TIMESTAMP WITHOUT TIME ZONE, "datetime_tz" TIMESTAMP WITH TIME ZONE, PRIMARY KEY ("name") )

DML

INSERT INTO foobar (name, date, datetime_notz, datetime_tz) VALUES ('foo', '2009-05-13', '2009-05-13T19:19:30.123456Z', '2009-05-13T19:19:30.123456Z'); INSERT INTO foobar (name, date, datetime_notz, datetime_tz) VALUES ('bar', '2009-05-13', '2009-05-13T19:19:30.123456+03:00', '2009-05-13T19:19:30.123456+03:00');

DQL

SELECT * FROM foobar;

+------+---------------+---------------+---------------+ | name | date | datetime_notz | datetime_tz | +------+---------------+---------------+---------------+ | foo | 1242172800000 | 1242242370123 | 1242242370123 | | bar | 1242172800000 | 1242242370123 | 1242231570123 | +------+---------------+---------------+---------------+

In this spirit, it is clear that there is a flaw, and the relevant spot can be discovered easily. 💥

sqlalchemy-cratedb/src/sqlalchemy_cratedb/dialect.py

Line 125 in df7fab9

return value.strftime('%Y-%m-%dT%H:%M:%S.%fZ')

145f701 aims to get the improvement straight, test outcomes now reflect the relevant conversion implicitly performed by CrateDB.

seut · 2024-06-25T09:19:15Z

tests/datetime_test.py

+INPUT_DATETIME_NOTZ = dt.datetime(2009, 5, 13, 19, 19, 30, 123456)
+INPUT_DATETIME_TZ = dt.datetime(2009, 5, 13, 19, 19, 30, 123456, tzinfo=CST())
+OUTPUT_DATE = INPUT_DATE
+OUTPUT_TIME = dt.time(19, 19, 30, 123000)
+OUTPUT_DATETIME_NOTZ = dt.datetime(2009, 5, 13, 19, 19, 30, 123000)
+OUTPUT_DATETIME_TZ = dt.datetime(2009, 5, 13, 19, 19, 30, 123000)


It looks a bit confusing that there isn't a difference when passing in a time zone or not. I guess this relates that when no time zone is given the local one which is also CST will be used in python? Inside CrateDB, all timestamps without timezone are treat as UTC. Shouldn't we then rather use an exotic one so the outputs aren't the same?

I will be happy to steer this patch into that direction. Thanks.

I guess this relates that when no time zone is given the local one which is also CST will be used in Python?

The root cause was different.

Co-authored-by: Marios Trivyzas <[email protected]>

... instead of `sa.DateTime` and `sa.TIMESTAMP`. Introduce `visit_TIMESTAMP` from PGTypeCompiler to render SQL DDL clauses like `TIMESTAMP WITH|WITHOUT TIME ZONE`.

Effectively, just add `%z` to the format string passed to `strftime`. -- https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes

amotl · 2024-06-25T13:50:13Z

tests/update_test.py

-        dt = datetime.strptime(args[1], '%Y-%m-%dT%H:%M:%S.%fZ')
+        dt = datetime.strptime(args[1], '%Y-%m-%dT%H:%M:%S.%f')


Can not use the %z suffix here. Otherwise:

ValueError: time data '2024-06-25T13:48:33.295461' does not match format '%Y-%m-%dT%H:%M:%S.%f%z'

amotl · 2024-06-25T13:50:31Z

tests/update_test.py

-        dt = datetime.strptime(args[2], '%Y-%m-%dT%H:%M:%S.%fZ')
+        dt = datetime.strptime(args[2], '%Y-%m-%dT%H:%M:%S.%f')


seut

👍 thx!

amotl changed the title ~~Types: Unlock supporting timezone-aware DateTime fields~~ [TYPES] Unlock supporting timezone-aware DateTime fields Jan 15, 2024

amotl changed the title ~~[TYPES] Unlock supporting timezone-aware DateTime fields~~ Types: Unlock supporting timezone-aware DateTime fields Jan 15, 2024

amotl force-pushed the amo/timezone-aware branch from 738fc7f to 2eca67a Compare January 15, 2024 23:07

amotl requested review from seut, matriv and surister June 22, 2024 20:38

amotl marked this pull request as ready for review June 22, 2024 20:39

amotl force-pushed the amo/timezone-aware branch 2 times, most recently from 4fb90ac to 347f2a7 Compare June 22, 2024 21:03

matriv reviewed Jun 24, 2024

View reviewed changes

amotl force-pushed the amo/timezone-aware branch 2 times, most recently from c7212b2 to 833328b Compare June 24, 2024 10:15

amotl requested a review from matriv June 24, 2024 10:18

amotl marked this pull request as draft June 24, 2024 12:46

amotl mentioned this pull request Jun 24, 2024

Types: Accept marshalling datetime.date values on DateTime fields #25

Merged

3 tasks

amotl force-pushed the amo/timezone-aware branch 3 times, most recently from 4482076 to 009d233 Compare June 24, 2024 18:27

seut reviewed Jun 25, 2024

View reviewed changes

amotl and others added 3 commits June 25, 2024 13:29

Types: Unlock supporting timezone-aware DateTime fields

6d5821d

Co-authored-by: Marios Trivyzas <[email protected]>

DateTime and more: Use CrateDB's DateTime

cfddf42

... instead of `sa.DateTime` and `sa.TIMESTAMP`. Introduce `visit_TIMESTAMP` from PGTypeCompiler to render SQL DDL clauses like `TIMESTAMP WITH|WITHOUT TIME ZONE`.

DateTime and more: Use time zone "Europe/Kyiv" for testing

619c991

amotl force-pushed the amo/timezone-aware branch from 009d233 to 619c991 Compare June 25, 2024 12:11

DateTime and more: Serialize CrateDB's DateTime types using zone info

145f701

Effectively, just add `%z` to the format string passed to `strftime`. -- https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes

amotl marked this pull request as ready for review June 25, 2024 12:58

amotl requested a review from seut June 25, 2024 12:58

amotl commented Jun 25, 2024

View reviewed changes

seut approved these changes Jun 25, 2024

View reviewed changes

amotl merged commit fb4c7be into main Jun 25, 2024
27 checks passed

amotl deleted the amo/timezone-aware branch June 25, 2024 14:02

amotl mentioned this pull request Jun 26, 2024

Return TIMESTAMP columns as native Python datetime objects to improve support for Apache Superset crate/crate-python#395

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Types: Unlock supporting timezone-aware `DateTime` fields #22

Types: Unlock supporting timezone-aware `DateTime` fields #22

amotl commented Jan 15, 2024 •

edited

Loading

matriv left a comment

matriv Jun 24, 2024

amotl Jun 24, 2024

matriv Jun 24, 2024

matriv Jun 24, 2024

amotl Jun 24, 2024

amotl Jun 25, 2024 •

edited

Loading

amotl Jun 25, 2024 •

edited

Loading

amotl Jun 25, 2024 •

edited

Loading

amotl Jun 25, 2024

amotl Jun 25, 2024

seut Jun 25, 2024

amotl Jun 25, 2024

amotl Jun 25, 2024

amotl Jun 25, 2024

amotl Jun 25, 2024

seut left a comment

		dt = datetime.strptime(args[1], '%Y-%m-%dT%H:%M:%S.%fZ')
		dt = datetime.strptime(args[1], '%Y-%m-%dT%H:%M:%S.%f')

		dt = datetime.strptime(args[2], '%Y-%m-%dT%H:%M:%S.%fZ')
		dt = datetime.strptime(args[2], '%Y-%m-%dT%H:%M:%S.%f')

Types: Unlock supporting timezone-aware DateTime fields #22

Types: Unlock supporting timezone-aware DateTime fields #22

Conversation

amotl commented Jan 15, 2024 • edited Loading

About

References

Backlog

matriv left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amotl Jun 25, 2024 • edited Loading

Choose a reason for hiding this comment

amotl Jun 25, 2024 • edited Loading

Choose a reason for hiding this comment

amotl Jun 25, 2024 • edited Loading

Choose a reason for hiding this comment

DDL

DML

DQL

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seut left a comment

Choose a reason for hiding this comment

Types: Unlock supporting timezone-aware `DateTime` fields #22

Types: Unlock supporting timezone-aware `DateTime` fields #22

amotl commented Jan 15, 2024 •

edited

Loading

amotl Jun 25, 2024 •

edited

Loading

amotl Jun 25, 2024 •

edited

Loading

amotl Jun 25, 2024 •

edited

Loading