Skip to content

Commit

Permalink
MongoDB: Configure MongoDBCrateDBConverter for updating commons-codec
Browse files Browse the repository at this point in the history
commons-codec 0.0.18 provides configuration capabilities about how to
decode timestamp values. CTK currently uses this configuration:

MongoDBCrateDBConverter(
    timestamp_to_epoch=True,
    timestamp_use_milliseconds=True,
)
  • Loading branch information
amotl committed Sep 25, 2024
1 parent d15c749 commit 67f5ed3
Show file tree
Hide file tree
Showing 3 changed files with 9 additions and 3 deletions.
1 change: 1 addition & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@


## Unreleased
- MongoDB: Configure `MongoDBCrateDBConverter` after updating to commons-codec 0.0.18

## 2024/09/22 v0.0.25
- Table Loader: Improved conditional handling of "transformation" parameter
Expand Down
9 changes: 7 additions & 2 deletions cratedb_toolkit/io/mongodb/copy.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,9 +57,14 @@ def __init__(
)
try:
transformation = tm.project.get(address=address)
logger.info(f"Applying transformation to: {address}")

Check warning on line 60 in cratedb_toolkit/io/mongodb/copy.py

View check run for this annotation

Codecov / codecov/patch

cratedb_toolkit/io/mongodb/copy.py#L60

Added line #L60 was not covered by tests
except KeyError:
pass
self.converter = MongoDBCrateDBConverter(transformation=transformation)
logger.warning(f"No transformation found for: {address}")
self.converter = MongoDBCrateDBConverter(

Check warning on line 63 in cratedb_toolkit/io/mongodb/copy.py

View check run for this annotation

Codecov / codecov/patch

cratedb_toolkit/io/mongodb/copy.py#L62-L63

Added lines #L62 - L63 were not covered by tests
timestamp_to_epoch=True,
timestamp_use_milliseconds=True,
transformation=transformation,
)
self.translator = MongoDBFullLoadTranslator(table_name=self.cratedb_table, converter=self.converter)

self.on_error = on_error
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ kinesis = [
"lorrystream[carabas]>=0.0.6",
]
mongodb = [
"commons-codec[mongodb,zyp]>=0.0.17",
"commons-codec[mongodb,zyp]>=0.0.18",
"cratedb-toolkit[io]",
"orjson<4,>=3.3.1",
"pymongo<5,>=3.10.1",
Expand Down

0 comments on commit 67f5ed3

Please sign in to comment.