feat: reference service 0.2.0

docs/migrating_to_16.md

@@ -11,16 +11,52 @@ Key points:
    * If needed, the default service specific volume directories defined in [bento.env](../etc/bento.env) can be overriden in your 
      `local.env`.
 * The WES schema has been updated, so the run database must be cleared.
+* The reference service has been updated, so an SQL migration must be run.
 
 
-## 1. Pre-update data config
+## 1. Run reference service pre-migration step
+
+```bash
+./bentoctl.bash shell reference-db
+PGPASSWORD="${POSTGRES_PASSWORD}"
+psql --user "${POSTGRES_USER}" reference
+```
+
+Then, begin a transaction:
+
+```sql
+BEGIN TRANSACTION;
+```
+
+Then, paste the contents of
+[migrate_v0_2.sql](https://github.com/bento-platform/bento_reference_service/blob/main/bento_reference_service/sql/migrate_v0_2.sql)
+from the reference service repository into the SQL command line.
+
+Next, commit the results of the transaction:
+
+```sql
+COMMIT;
+```
+
+Finally, exit the shell (via successive `^D`).
+
+
+## 2. Stop Bento
+
+```bash
+./bentoctl.bash stop
+```
+
+
+## 3. Pre-update data config
 
 Before updating, perform the following steps:
 
 * Shut down Bento with `./bentoctl.bash stop`
 * In `local.env`, replace `BENTOV2_ROOT_DATA_DIR` with `BENTO_FAST_DATA_DIR` and `BENTO_SLOW_DATA_DIR`
 
-To minimize side effects in local environments, we recommend that you use the same directory as before for both new variables.
+To minimize side effects in local environments, we recommend that you use the same directory as before for both new 
+variables.
 
 ```bash
 # local.env
@@ -33,28 +69,30 @@ BENTO_FAST_DATA_DIR=some_dir
 BENTO_SLOW_DATA_DIR=some_dir
 ```
 
-*IMPORTANT:* If you decide to split the data in distinct directories, make sure to migrate the data accordingly.
+*IMPORTANT:* If you decide to split the data in distinct directories, **make sure to migrate the data accordingly**.
 
 
-## 2. Update images
+## 4. Update images
 
 ```bash
 ./bentoctl.bash pull
 ```
 
 
-## 3. Create new Docker volume directories
-
-This step is optional if you DIDN'T split the data directories in step 1.
+## 5. Create new Docker volume directories
 
 ```bash
-./bentoctl.bash init-dirs  # new volume directory for DRS temporary files
-
-# migrate the old data
+./bentoctl.bash init-dirs  # new volume directory for DRS and Reference temporary files
 ```
 
 
-## 4. Restart Bento
+## 6. Migrate data from old `BENTOV2_ROOT_DATA_DIR` location to `BENTO_FAST_DATA_DIR`/`BENTO_SLOW_DATA_DIR`
+
+This is only needed if the data was split into distinct directories in part 1. All service data should be copied into 
+the corresponding locations as specified in `etc/bento.env`.
+
+
+## 7. Restart Bento
 
 ```bash
 ./bentoctl.bash start

etc/bento.env

@@ -179,14 +179,15 @@ BENTOV2_EVENT_RELAY_CPUS=1
 # Reference
 #  - Service
 BENTO_REFERENCE_IMAGE=ghcr.io/bento-platform/bento_reference_service
-BENTO_REFERENCE_VERSION=0.1.2
+BENTO_REFERENCE_VERSION=0.2.0
 BENTO_REFERENCE_VERSION_DEV=${BENTO_REFERENCE_VERSION}-dev
 BENTO_REFERENCE_CONTAINER_NAME=${BENTOV2_PREFIX}-reference
 BENTO_REFERENCE_NETWORK=${BENTOV2_PREFIX}-reference-net
 BENTO_REFERENCE_INTERNAL_PORT=${BENTO_STD_SERVICE_INTERNAL_PORT}
 BENTO_REFERENCE_EXTERNAL_PORT=9510
 BENTO_REFERENCE_DEBUGGER_INTERNAL_PORT=9511
 BENTO_REFERENCE_DEBUGGER_EXTERNAL_PORT=9511
+BENTO_REFERENCE_TMP_VOL_DIR=${BENTO_FAST_DATA_DIR}/reference/tmp
 #  - Database
 BENTO_REFERENCE_DB_IMAGE=postgres
 BENTO_REFERENCE_DB_VERSION=16

lib/reference/docker-compose.reference.yaml

@@ -15,6 +15,9 @@ services:
       - SERVICE_URL_BASE_PATH=${BENTOV2_PUBLIC_URL}/api/reference
       - CORS_ORIGINS=${BENTO_CORS_ORIGINS}
       - BENTO_AUTHZ_SERVICE_URL
+      - FILE_INGEST_TMP_DIR=/reference/tmp
+    volumes:
+      - ${BENTO_REFERENCE_TMP_VOL_DIR}:/reference/tmp
     healthcheck:
       test: [ "CMD", "wget", "-q0-", "http://localhost:${BENTO_REFERENCE_INTERNAL_PORT}/service-info", "&&", "echo" ]
       timeout: ${BENTOV2_HEALTHCHECK_TIMEOUT}

py_bentoctl/other_helpers.py

@@ -212,6 +212,7 @@ def init_dirs():
         "katsu-db": "BENTOV2_KATSU_DB_PROD_VOL_DIR" if c.DEV_MODE else "BENTOV2_KATSU_DB_DEV_VOL_DIR",
         "notification": "BENTOV2_NOTIFICATION_VOL_DIR",
         "redis": "BENTOV2_REDIS_VOL_DIR",
+        "reference": "BENTO_REFERENCE_TMP_VOL_DIR",
         "reference-db": "BENTO_REFERENCE_DB_VOL_DIR",
         "wes": "BENTOV2_WES_VOL_DIR",
         "wes-tmp": "BENTOV2_WES_VOL_TMP_DIR",