Skip to content

Data models for transforming the raw data from Couch DB into models that can then be queried to build dashboards

License

Notifications You must be signed in to change notification settings

medic/cht-pipeline

Repository files navigation

A set of SQL queries that transform raw CouchDB data into a more useful format for analytics. It uses dbt to define the models that are translated into PostgreSQL tables or views, which makes it easier to query the data in the analytics platform of choice.

Setup

Follow the instructions in the CHT Sync documentation to set up the data pipeline.

Run dbt models unit tests locally

Prerequisites

  • Docker

Run the tests

  1. Navigate to tests folder.
  2. Run the test script:
# set environment variables, install dbt dependencies, seed data, run dbt, run test
./run_dbt_tests.sh

Release Process

This repo has an automated release process where each feature/bug fix will be released immediately after it is merged to main. The release type is determined by the commit message format. Have a look at the development workflow in the Contributor Handbook for more information.

Commit message format

The commit format should follow the convention outlined in the CHT docs. Examples are provided below.

Type Example commit message Release type
Bug fixes fix(#123): rename column names patch
Performance perf(#789): add new indexes patch
Features feat(#456): add new model minor
Non-code chore(#123): update README none
Breaking perf(#2): remove data_record model
BREAKING CHANGE: form models should now read from new_model
major

About

Data models for transforming the raw data from Couch DB into models that can then be queried to build dashboards

Resources

License

Stars

Watchers

Forks