added initial files for qc #39

shanice-skylight · 2025-01-30T23:32:47Z

This PR includes:

A docker-compose file to deploy the following containers: database, application, and Portainer.
The vs_dump.sql file, which initializes all value sets, concepts, and foreign key mappings extracted from a fresh eRSD pull and processed through our creation functions.
Environment variable files for both the application and database containers.
A provisioning script (qc-provision.sh) specific to query connector.
A Packer file tailored for query connector.

In the future, the separate provisioning and Packer files can be consolidated into a single file, leveraging GitHub workflows to determine which application to build the virtual image for.

shanice-skylight · 2025-02-11T04:26:23Z

work in progress on PR

alismx

I plan to run a build from this branch shortly, but I wanted to post my initial questions and comments!

alismx · 2025-02-24T19:35:56Z

docker/query-connector/vs_dump.sql

How long does the db-creation script take?

Is there another place this dump is stored, like in the query connector repo? If so, we could reference the raw file in that repo.

It doesn't take long, the dump is also stored in the dibbs-query-connector

@rin-skylight @shanice-skylight We aren't going to need the dump in this repo because the provision.sh script clones the dibbs-query-connector repo since it's in that repo as linked; we need to make sure the docker file mount source is correct.

@alismx Can you reference where in the script it clones dibbs-query-connector, I can't find it?

docker/query-connector/README.md

docker/query-connector/db.env

alismx · 2025-02-24T19:42:55Z

docker/query-connector/docker-compose.yml

@@ -0,0 +1,49 @@
+services:
+  # PostgreSQL DB for custom query and value set storage
+  db:


Is this required for the build to succeed? Is the database going to be bundled into the VM for partners to launch that way? My understanding was that we expect a partner to launch a database somewhere(RDS), and they would configure the app to connect to that.

The VM includes a database preloaded with all value sets, concepts, and foreign key mappings extracted from a fresh eRSD pull and processed through our creation functions. We anticipate that partners will eventually connect their own database.

I think it's worth discussing whether the database is fully necessary for this VM. We're trying to keep the appliance as lightweight as possible, and it has a tiny disk size of only 8 GB. If we expect a partner to connect their own DB, we run the risk of our bundled postgres instance becoming a maintenance and security issue by just hanging out. There's a chance it will be reintroduced by new compose files on a version update...would a partner want that if they've already cut it out of their build?

I think for now, we should bundle it in for partners, but it's certainly possible they will want to use their own.

I looked at the dump file, and it's a little over 8M, but I don't know how much storage the database will need or how large it will grow. I've included a screenshot of the VM as it started. We'll need to consider storage for this and any other data/logging that is held in the VM.

I'm not sure we can determine the final size of this database at the moment. Regarding the ERSD and UMLS codes, they are updated once or twice a year. @nickclyde I suggest treating this as a research or brainstorming topic, but for now, we could leave it as is. Here's the ticket for it.

alismx · 2025-02-24T21:40:04Z

packer/ubuntu-server/qc-ubuntu.pkr.hcl

+  provisioner "shell" {
+    only = ["qemu.iso"]
+    scripts = [
+        "scripts/provision.sh"


When I ran the build it ran the provision script setup for the ecr-viewer.

Good catch,thanks

EmmanuelNwa247

you might need to change line 90 "scripts/provision.sh" in your qc-ubuntu.pkr.hcl file to reference your new changes "scripts/qc-provision.sh"

EmmanuelNwa247

LGTM!!!

rin-skylight

This is a great start, but I would like to make sure we don't introduce duplicate code. I think this PR should integrate the variable-based changes that Alis demonstrated for the team in their most recent PR. If this one needs to wait for a bit, let's do that. Getting it done right is more important than speed.

rin-skylight · 2025-02-27T19:37:47Z

docker/query-connector/docker-compose.yml

@@ -0,0 +1,49 @@
+services:
+  # PostgreSQL DB for custom query and value set storage
+  db:


I think it's worth discussing whether the database is fully necessary for this VM. We're trying to keep the appliance as lightweight as possible, and it has a tiny disk size of only 8 GB. If we expect a partner to connect their own DB, we run the risk of our bundled postgres instance becoming a maintenance and security issue by just hanging out. There's a chance it will be reintroduced by new compose files on a version update...would a partner want that if they've already cut it out of their build?

rin-skylight · 2025-02-27T19:38:39Z

docker/query-connector/docker-compose.yml

+
+  # Next.js app
+  query-connector:
+    image: ghcr.io/cdcgov/dibbs-query-connector/query-connector:latest


Rather than using latest, we will need to pin this to a specific version. This prevents the stack from auto-updating, and allows us to build specific versions.

Currently we only tag the packages with latest and main tags so I added a ticket to update the workflow to add incremental versions.

docker/query-connector/docker-compose.yml

rin-skylight · 2025-02-27T19:39:41Z

docker/query-connector/query-connector.env

@@ -0,0 +1,2 @@
+NODE_ENV=production
+DATABASE_URL=postgres://postgres:pw@db:5432/qc_db


We should revisit the DATABASE_URL value if we expect customers to attach their own DB.

Currently its an option for customers to attach their own DB but its not a requirement.

packer/ubuntu-server/http/user-data

packer/ubuntu-server/qc-ubuntu.pkr.hcl

nickclyde

Thanks Shanice!!

nickclyde · 2025-02-28T00:31:49Z

docker/query-connector/docker-compose.yml

@@ -0,0 +1,49 @@
+services:
+  # PostgreSQL DB for custom query and value set storage
+  db:


I think for now, we should bundle it in for partners, but it's certainly possible they will want to use their own.

alismx added this to the DIBBs-In-A-Box! milestone Jan 31, 2025

alismx added the Cloud Enablement Cloud Enablement DevOps label Jan 31, 2025

alismx assigned shanice-skylight Jan 31, 2025

shanice-skylight mentioned this pull request Feb 11, 2025

Add compose file for Query Connector #18

Open

shanice-skylight marked this pull request as ready for review February 20, 2025 04:20

shanice-skylight requested review from rin-skylight, alismx and EmmanuelNwa247 February 20, 2025 04:22

alismx reviewed Feb 24, 2025

View reviewed changes

EmmanuelNwa247 reviewed Feb 26, 2025

View reviewed changes

shanice-skylight requested review from alismx and EmmanuelNwa247 February 27, 2025 16:21

EmmanuelNwa247 reviewed Feb 27, 2025

View reviewed changes

rin-skylight requested changes Feb 27, 2025

View reviewed changes

nickclyde approved these changes Feb 28, 2025

View reviewed changes

shanice-skylight added 11 commits March 4, 2025 19:47

added initial files for qc

375d310

modified script just for query connector

c61dd11

removed keycloak and added portainer

cdf8fb3

removed keycloak dependency

8a8cea7

added restart to db and query connector

4feefea

moved env to env files

b3dc9bc

added dump file, documenation for it, and db migration cmds

a24d532

removed flway and modified query connector command

0b9172a

added a separate provision script and pkr file for each app

6062b6d

added review updates

4ccf2f5

removed keycloak and authentication from env.sample file

8109c6a

shanice-skylight force-pushed the shanice/query-connector branch from 9ef386a to 8109c6a Compare March 5, 2025 00:58

removed redundancy

8adb977

removed typo

e1ade4b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

added initial files for qc #39

added initial files for qc #39

shanice-skylight commented Jan 30, 2025 •

edited

Loading

shanice-skylight commented Feb 11, 2025

alismx left a comment

alismx Feb 24, 2025

shanice-skylight Feb 27, 2025

alismx Feb 28, 2025

shanice-skylight Mar 5, 2025

alismx Feb 24, 2025

shanice-skylight Feb 27, 2025

rin-skylight Feb 27, 2025

nickclyde Feb 28, 2025

alismx Feb 28, 2025 •

edited

Loading

shanice-skylight Mar 5, 2025 •

edited

Loading

alismx Feb 24, 2025

shanice-skylight Feb 27, 2025

EmmanuelNwa247 left a comment

EmmanuelNwa247 left a comment

rin-skylight left a comment

rin-skylight Feb 27, 2025

rin-skylight Feb 27, 2025

shanice-skylight Mar 5, 2025

rin-skylight Feb 27, 2025

shanice-skylight Mar 5, 2025

nickclyde left a comment

nickclyde Feb 28, 2025

		@@ -0,0 +1,2 @@
		NODE_ENV=production
		DATABASE_URL=postgres://postgres:pw@db:5432/qc_db

added initial files for qc #39

Are you sure you want to change the base?

added initial files for qc #39

Conversation

shanice-skylight commented Jan 30, 2025 • edited Loading

shanice-skylight commented Feb 11, 2025

alismx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alismx Feb 28, 2025 • edited Loading

Choose a reason for hiding this comment

shanice-skylight Mar 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EmmanuelNwa247 left a comment

Choose a reason for hiding this comment

EmmanuelNwa247 left a comment

Choose a reason for hiding this comment

rin-skylight left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nickclyde left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shanice-skylight commented Jan 30, 2025 •

edited

Loading

alismx Feb 28, 2025 •

edited

Loading

shanice-skylight Mar 5, 2025 •

edited

Loading