Code

feat: run dbt in batches#158

Code

Open

njuguna-n wants to merge 12 commits intomainmedic/cht-sync:mainfrom 156-run-dbt-in-batchesmedic/cht-sync:156-run-dbt-in-batchesCopy head branch name to clipboard

+442-146

Commits

Commits on Sep 25, 2024

• feat: process initial sync in batches

  njogz

  committed
  f16ecf8

  Copy full SHA for f16ecf8

Commits on Sep 27, 2024

• refactor: try alternative batching process

  njogz

  committed
  93d67f2

  Copy full SHA for 93d67f2

Commits on Sep 30, 2024

• refactor: how we do batch dbt updates

  njogz

  committed
  4314faa

  Copy full SHA for 4314faa

Commits on Oct 4, 2024

• feat: run dbt in batch mode only

  njogz

  committed
  49c5cc6

  Copy full SHA for 49c5cc6

• chore: run e2e test in batches

  njogz

  committed
  e1e3b9c

  Copy full SHA for e1e3b9c

• feat: run in batches or regular run

  njogz

  committed
  a520e30

  Copy full SHA for a520e30

• chore: use variable

  njogz

  committed
  8bd222b

  Copy full SHA for 8bd222b

Commits on Oct 7, 2024

• refactor: handle batch update error and success in different functions

  njogz

  committed
  0f3405f

  Copy full SHA for 0f3405f

• chore: refactor: use BATCH_STATUS_TABLE variable and remove unused function

  njogz

  committed
  b5522c9

  Copy full SHA for b5522c9

Commits on Oct 8, 2024

• feat: check if metadata table exists

  njogz

  committed
  2caf09d

  Copy full SHA for 2caf09d

• chore: chnge where the table check happens

  njogz

  committed
  c36ced3

  Copy full SHA for c36ced3

Commits on Oct 9, 2024

• feat: rename pythin file, cchange setup function to use one connection, and remove document_matadata table check

  njogz

  committed
  4055e3f

  Copy full SHA for 4055e3f