[SHEP-135] Add automatic generation of dbml file from schema on migra…

…tions (#300)

## References

[SHEP-135](https://mozilla-hub.atlassian.net/browse/SHEP-135)

## Problem Statement

We would like a way to better document the "total truth state" of our
database after each migration is run. To do this, we propose dumping a
`.dbml` file after each migration for better transparency on the total
state of the database.

## Proposed Changes

Two new steps have been added to the `make migrate` command. Namely:
1. After `python manage.py migrate`, a `pg_dump` command is run which
saves the `.sql` representation of our database's schema to a schema.sql
file.
2. After the `.sql` dump completes, we convert the sql to a dbml which
is more friendly for humans to read, and can be easily dropped into
[dbdiagram](https://dbdiagram.io/) for easy visualization.

## Verification Steps

New dependencies required for running `sql2dbml` should be included as
dev dependencies in the `package.json` file in the root directory.
Verify the following:
1. `make install` correctly identifies the dev dependency and installs
it
2. `make migrate` operates as normally expected _and_ a new file appears
in the root directory named `schema.dmbl`
3. `make makemigrations` operates as normally expected and has the same
behavior as 2. above.

[SHEP-135]:
https://mozilla-hub.atlassian.net/browse/SHEP-135?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

.gitignore

@@ -26,6 +26,7 @@ share/python-wheels/
 .installed.cfg
 *.egg
 MANIFEST
+node_modules/
 
 # PyInstaller
 #  Usually these files are written by a python script from a template

Makefile

@@ -111,15 +111,20 @@ local-test: local-test-django local-test-react   ##  Run all tests when developi
 makemigrations-empty: ##  Create an empty migrations file for manual migrations
 	docker exec -it consvc-shepherd-app-1 python manage.py makemigrations --empty consvc_shepherd
 
-migrate: ##  Run migrate on the docker container
+migrate: ##  Run migrate on the docker container. This will run Django migrations and create .sql, .dbml, and visual files
+	npm list -g @dbml/cli || npm install -g @dbml/cli
 	docker exec -it consvc-shepherd-app-1 python manage.py migrate
+	docker exec -it consvc-shepherd-db-1 pg_dump -U postgres -s -F p -E UTF-8  postgres > schema.sql
+	sql2dbml schema.sql --postgres -o schema.dbml
+	rm schema.sql
+	@echo "Migrations complete!"
 
 makemigrations: ##  Run makemigrations on the docker container set MIGRATE=false prevent automatic migration.
 	@echo "Making migrations..."
 	docker exec -it consvc-shepherd-app-1 python manage.py makemigrations
 	@if [ "$(MIGRATE)" = "true" ]; then \
 		echo "Applying migration..."; \
-		docker exec -it consvc-shepherd-app-1 python manage.py migrate; \
+		make migrate; \
 	fi
 
 ruff: install ##  **Experimental** Run ruff linter. To fix and format files.

docs/quickStart.md

@@ -1,6 +1,7 @@
 # Quick Start Development Guide
 
 Shepherd is a grab-bag of Ad Operations tools, and currently consists of
+
 * `consvc_shepherd`: A full-stack Python Django app, as well as a json REST API
 * `contile`: A Python Django app
 * `ad-ops-dashboard`: A Node/Typescript React app, backed by the `consvc_shepherd` REST API
@@ -10,12 +11,14 @@ Shepherd is a grab-bag of Ad Operations tools, and currently consists of
 To use consvc-shepherd, you'll need a Python 3.11 development environment with Poetry installed and Docker.
 
 It is recommended to use `pyenv` and the `pyenv-virtualenv` plugin for your virtual environment.
+
 1. Install `pyenv` using the [latest documentation](https://github.com/pyenv/pyenv#installation) for your platform.
 1. Follow the instructions to install the `pyenv-virtualenv` plugin.
 See the [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) documentation.
 1. Ensure you've added `pyenv` and `pyenv-virtualenv` to your `PATH`:
 
     Ex:
+
     ```shell
     export PATH="$HOME/.pyenv/bin:$PATH"
     eval "$(pyenv init -)"
@@ -55,13 +58,22 @@ See the [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) documentat
 1. Install your dependencies:
 
     ```shell
-    $ pip install poetry
-    $ poetry install
+    pip install poetry
+    poetry install
+    ```
+
+    We also require a global dependency to be installed now in order to make migration diagrams automagically in `dbml format.`
+
+    ```shell
+    npm install -g @dbml/cli
     ```
 
+    **Note** that this requires Node to be already installed on your computer. We recommend using something like `nvm`. See the section below on Node setup for more information on how to do that.
+
 1. [Install Docker](https://docs.docker.com/engine/install/), if not already installed.
 
 1. Build the Docker image and start the container:
+
     ```shell
     docker compose up --build
     ```
@@ -71,12 +83,19 @@ See the [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) documentat
 1. Create database migrations and run migrations.
 
     You may have to do this periodically as you modify or create models. Shell in as above and run the following commands:
+
     ```shell
     docker exec -it consvc-shepherd-app-1 bash # can also be run with `make debug`
     python manage.py makemigrations
-    python manage.py migrate
+    make migrate   # Applies the migrations and generates schema.dbml
     ```
 
+    If this completes successfully, you should see a "Migrations complete!" message in your console.
+
+   ##### A note about .dbml
+
+    In addition to migrations being performed when running this make command, you should also see a `schema.dbml` file in the root directory. This file is part of an effort to have self-documenting migrations where a visual schema diagram will be automatically built after each migration. This file *should* be included in source control. For now, this file must be manually imported into [dbdiagram](https://dbdiagram.io/home) to automagically show a visual representation of the new schema. Eventually we'd like to make that process automatic as well. [SHEP-163](https://mozilla-hub.atlassian.net/browse/SHEP-163). For more documentation on `dbml`, see [these docs](https://dbml.dbdiagram.io/home/).
+
 1. Connect to the DB with `psql`.
 
     While the Django Admin interface at `/admin` shows all the data in our DB in a human readable way, it may sometimes be
@@ -91,16 +110,19 @@ See the [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) documentat
 1. **(Optional)** Seed Shepherd DB with random test data. This is useful when testing schema changes without having to run our sync scripts to get real data.
 
     Shell into the `consvc-shepherd-app-1` container as mentioned above and run the following command:
+
     ```shell
     python manage.py seed
     ```
 
     The script will output a success message once the database has been seeded:
+
     ```shell
     Successfully seeded the database with random data
     ```
 
     Shell into `psql` and describe any of the DB tables (see example below), you should see that it contains some random data. Alternatively, you can look at the data by accessing the admin console at [http://0.0.0.0:7001/admin](http://0.0.0.0:7001/admin).
+
     ```shell
     SELECT * FROM consvc_shepherd_deliveredflight;
     ```
@@ -137,26 +159,28 @@ It is recommended to use a node version manager to allow for multiple `node`s on
 3. Install the `node` version specified in `ad-ops-dashboard/.nvmrc`:
 
     ```shell
-    $ nvm install
+    nvm install
     ```
 
 4. Install `ad-ops-dashboard` app's dependencies:
 
     ```shell
-    $ npm install
+    npm install
     ```
+
 5. Set up the `ad-ops-dashboard` environment:
 
     ```shell
-    $ cp .env.example .env
+    cp .env.example .env
     ```
+
 6. Spin up the `ad-ops-dashboard` app and REST api together:
 
     ```shell
-    $ cd ..
-    $ docker-compose up --build
+    cd ..
+    docker-compose up --build
     ```
 
     (You can also run the `ad-ops-dashboard` without the API via `npm run dev` from `./ad-ops-dashboard`)
 
-7. Visit the home page at http://0.0.0.0:5173/
+7. Visit the home page at <http://0.0.0.0:5173/>