dbt-sts-ge

Playground pipeline with Airflow 2, Spark Thrift Server, dbt and Great Expectations

How to run

First of all you need to build the required Airflow and Spark Thrift Server containers.
- Go to the folder dockers and run the script called build_dockers.sh.
- Back to the main folder.
Run docker compose with the command: docker-compose -f docker-compose.yaml up
- Airflow should be listening in the http address: http://localhost:8080/
- Sparkt Thrift Server should be listening in the thrift address: jdbc:hive2://localhost:10000

Airflow is running with the default username and password whose value is airflow.