Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add person distinct ID overrides squash job (as dagster job) #27710

Merged
merged 45 commits into from
Jan 22, 2025
Merged
Show file tree
Hide file tree
Changes from 43 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
b75e105
wip
tkaemming Jan 15, 2025
00b4c82
more queries
tkaemming Jan 16, 2025
75c359e
mutation queries
tkaemming Jan 16, 2025
4568d19
lots of stuff missing but runs happy path
tkaemming Jan 16, 2025
8701ab4
use run id for initial snapshot table creation
tkaemming Jan 16, 2025
16e3c1a
require the timestamp to be provided as config
tkaemming Jan 16, 2025
22a0d32
split create and populate steps
tkaemming Jan 16, 2025
0cd5e70
much refactoring
tkaemming Jan 16, 2025
5f36948
clumsy mutation waiter
tkaemming Jan 16, 2025
418e96a
more clumsy
tkaemming Jan 16, 2025
a055f0b
syntax
tkaemming Jan 16, 2025
1870834
easy enough
tkaemming Jan 17, 2025
907f772
split up more things that should be able to fail/retry independently
tkaemming Jan 17, 2025
3ac4798
clarify naming
tkaemming Jan 17, 2025
004573a
fold some excessively long lines
tkaemming Jan 17, 2025
0111a18
take a pass through to try and determine what needs to happen in case…
tkaemming Jan 17, 2025
25e7e29
clean up how mutations are waited
tkaemming Jan 17, 2025
0c731b8
not sure this is the best logging approach but it'll do for now
tkaemming Jan 17, 2025
4fdecfd
timestamp can actually just be a format that is parseable by clickhouse
tkaemming Jan 17, 2025
6ffa346
clean up dictionary loading
tkaemming Jan 17, 2025
2f9b3f5
make it safe to run enqueue operations multiple times
tkaemming Jan 17, 2025
748fc56
play it safe in populate
tkaemming Jan 17, 2025
acb0ace
formatting
tkaemming Jan 17, 2025
4eca7e0
cluster as resource
tkaemming Jan 17, 2025
35a866e
shards as config
tkaemming Jan 17, 2025
bfea2c5
configurable max execution time for dictionary creation
tkaemming Jan 17, 2025
3672289
support passing settings through config
tkaemming Jan 17, 2025
232a3a4
very permissive default settings
tkaemming Jan 17, 2025
6bdd544
docstrings
tkaemming Jan 17, 2025
a6b5d27
tidy
tkaemming Jan 17, 2025
328f8ad
make it so that clickhouse database management is a bit easier to use…
tkaemming Jan 18, 2025
a8dd161
"a test"
tkaemming Jan 18, 2025
15389b8
test cleanup actions
tkaemming Jan 18, 2025
ed86838
Merge branch 'master' into squash-dag
tkaemming Jan 20, 2025
e2dd839
fixtures
tkaemming Jan 20, 2025
88b4539
add test data
tkaemming Jan 20, 2025
9df49f4
actually assert some things
tkaemming Jan 20, 2025
2181485
expand test
tkaemming Jan 20, 2025
cb86cb2
more tidy
tkaemming Jan 20, 2025
55bfa23
move get_cluster to a better home
tkaemming Jan 20, 2025
1907cb9
Update query snapshots
github-actions[bot] Jan 21, 2025
0817855
be less clever with typing
tkaemming Jan 21, 2025
b18ccf7
no `Mapping.__or__` support so just do the other thing
tkaemming Jan 21, 2025
cce62ff
consolidate mostly duplicate _find_existing_mutation definitions
tkaemming Jan 21, 2025
e1ae63f
Merge branch 'master' into squash-dag
tkaemming Jan 22, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions dags/definitions.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,14 @@
from dagster import Definitions, load_assets_from_modules

from . import ch_examples, deletes, orm_examples
from .person_overrides import ClickhouseClusterResource, squash_person_overrides

all_assets = load_assets_from_modules([ch_examples, deletes, orm_examples])

defs = Definitions(
assets=all_assets,
jobs=[squash_person_overrides],
resources={
"cluster": ClickhouseClusterResource.configure_at_launch(),
},
)
Loading
Loading