feat!: authorized Katsu endpoints + slightly auth-aware Beacon

README.md

@@ -44,6 +44,7 @@ that make up the Bento platform.
 
 ### Migration documents
 
+* [v16 to v17](./docs/migrating_to_17.md)
 * [v15.2 to v16](./docs/migrating_to_16.md)
 * [v15.1 to v15.2](./docs/migrating_to_15_2.md)
 * [v15 to v15.1](./docs/migrating_to_15_1.md)

docs/installation.md

@@ -276,8 +276,10 @@ specified in the step above.
 ./bentoctl.bash init-auth
 ```
 
-**If using an external identity provider**, only start the cluster's gateway
-after setting `CLIENT_SECRET` in your local environment file:
+After running `init-auth`, be sure to put all client secrets into your `local.env` file!
+
+**If using an external identity provider**, only start the cluster's gateway after setting various `*_CLIENT_SECRET` 
+variables in your local environment file:
 
 ```bash
 ./bentoctl.bash run gateway
@@ -297,7 +299,7 @@ utilize new variables generated during the OIDC configuration.
 
 ## 6. Configure permissions
 
-### a. Create superuser permissions in the new Bento authorization service
+### a. Create superuser permissions in the Bento authorization service
 
 First, run the authorization service and then open a shell into the container:
 
@@ -317,24 +319,54 @@ which in Keycloak should be a UUID.
 
 ### b. Create grants for the Workflow Execution Service (WES) OAuth2 client
 
-Run the following commands to set up authorization for the WES client. Don't forget to replace `ISSUER_HERE` by the 
-issuer URL!
+Run the following commands to set up authorization for the WES client. 
+**Don't forget to replace `<ISSUER_HERE>` with the issuer URL!**
 
 ```bash
 # This grant is a temporary hack to get permissions working for v12/v13. In the future, it should be removed.
 bento_authz create grant \
-  '{"iss": "ISSUER_HERE", "client": "wes"}' \
+  '{"iss": "<ISSUER_HERE>", "client": "wes"}' \
   '{"everything": true}' \
   'view:private_portal'
 
 # This grant gives permission to access and ingest data into all projects and the reference genome service
 bento_authz create grant \
-  '{"iss": "ISSUER_HERE", "client": "wes"}' \
+  '{"iss": "<ISSUER_HERE>", "client": "wes"}' \
   '{"everything": true}' \
   'query:data' 'ingest:data' 'ingest:reference_material' 'delete:reference_material'
 ```
 
-### c. *Optional step:* Assign portal access to all users in the instance realm
+### c. Create a grant for the aggregation and Beacon services
+
+Run the following commands to set up authorization for the aggregation/Beacon client. 
+**Don't forget to replace `<ISSUER_HERE>` with the issuer URL!**
+
+```bash
+# In the future, view:private_portal will need to be removed from this grant.
+bento_authz create grant \
+  '{"iss": "<ISSUER_HERE>", "client": "aggregation"}' \
+  '{"everything": true}' \
+  'query:data' 'view:private_portal'
+```
+
+
+### d. Configure public data access for all users, including anonymous visitors (if desired):
+
+To configure public data access, run the following command in the authorization service container. Note that with the
+`full` value, **THIS GIVES FULL DATA ACCESS TO EVERYONE WHO VISITS YOUR INSTANCE!**
+
+```bash
+# Configure public data access
+# ----------------------------
+# The level below ("counts") preserves previous functionality. Other possible options are:
+#  - none - will do nothing.
+#  - bool - for censored true/false discovery, but in effect right now forbids access.
+#  - counts - for censored count discovery.
+#  - full - allows full data access (record-level, including sensitive data such as IDs), uncensored counts, etc.
+bento_authz public-data-access counts
+```
+
+### e. Assign portal access to all users in the instance realm
 
 We added a special permission, `view:private_portal`, to Bento v12/v13 in order to carry forward the current
 'legacy' authorization behaviour for one more major version. This permission currently behaves as a super-permission,

docs/migrating_to_17.md

@@ -1,6 +1,91 @@
 # Migrating to Bento v17
 
 Key points:
-* Bento now has observability tools to help monitor the services (Grafana)
-<!-- TODO: add migration guide for Grafana, with examples of user role attribution -->
+
+* Bento now has observability tools to help monitor the services (Grafana). Some setup is required for this feature to 
+  work.
+* Katsu discovery endpoints now have an authorization layer. 
+  * Data that used to be completely public by default (i.e., 
+    censored counts) now requires a permission (`query:project_level_counts` and/or `query:dataset_level_counts`), and 
+    thus a grant in the authorization service. 
+  * Beacon now requires a client ID/secret and an authorization service grant to access uncensored data.
+* Katsu discovery is now more granular, and can be configured to the project or dataset level, in addition to the 
+  instance level. See the [Public data discovery configuration](./public_discovery.md) document for more information.
 * ...
+
+
+## 1. Stop Bento
+
+```bash
+./bentoctl.bash stop
+```
+
+
+## 2. Update images
+
+```bash
+./bentoctl.bash pull
+```
+
+
+## 3. Set up credentials for aggregation/Beacon and, optionally, set up Grafana
+
+If you wish to enable Grafana, you first must enable the monitoring feature in your `local.env` file:
+
+```bash
+BENTO_MONITORING_ENABLED='true'
+```
+
+To create the client secrets for aggregation/Beacon and Grafana (if the latter is enabled), run the following commands:
+
+```bash
+./bentoctl.bash start auth
+./bentoctl.bash init-auth
+```
+
+**Reminder:** Make sure to put the client secret(s) generated by `init-auth` into your `local.env` file!
+
+Aggregation/Beacon data access authorization will not work until an authorization service grant is configured; 
+see step 4 below.
+
+
+## 4. Set up aggregation/Beacon permissions and public data access grants
+
+Now that Beacon uses a client ID/secret to get authorized, uncensored data access for discovery, a grant must be 
+configured to give the aggregation/Beacon client data access.
+
+Another change to permissions: starting from Bento v17, anonymous visitors do not have access to see censored counts 
+data by default, even if a discovery configuration has been set up. For anonymous visitors to access data, a level 
+(`bool`, `counts`, `full`) must be chosen and passed to the `bento_authz` CLI command below.
+
+```bash
+./bentoctl.bash shell authz
+
+# Configure aggregation/Beacon permissions
+# ----------------------------------------
+# This assumes the aggregation/Beacon client ID is "aggregation". 
+# <ISSUER_HERE> MUST be replaced with your actual issuer value.
+#  - The query:data permission gives access to Katsu endpoints which are properly authz-enabled.
+#  - The view:private_portal permission gives access to Katsu and Gohan endpoints where the proxy still manages access.
+#    This permission will be removed in an uncoming version.
+bento_authz create grant \
+  '{"iss": "<ISSUER_HERE>", "client": "aggregation"}' \
+  '{"everything": true}' \
+  'query:data' 'view:private_portal'
+
+# Configure public data access
+# ----------------------------
+# The level below ("counts") preserves previous functionality. Other possible options are:
+#  - none - will do nothing.
+#  - bool - for censored true/false discovery, but in effect right now forbids access.
+#  - counts - for censored count discovery.
+#  - full - allows full data access (record-level, including sensitive data such as IDs), uncensored counts, etc.
+bento_authz public-data-access counts
+```
+
+
+## 5. Start Bento
+
+```bash
+./bentoctl.bash start
+```

etc/bento.env

@@ -31,7 +31,7 @@ BENTOV2_GATEWAY_INTERNAL_CERTS_DIR=/usr/local/openresty/nginx/certs
 
 # Gateway
 BENTOV2_GATEWAY_IMAGE=ghcr.io/bento-platform/bento_gateway
-BENTOV2_GATEWAY_VERSION=0.12.0
+BENTOV2_GATEWAY_VERSION=edge
 BENTOV2_GATEWAY_VERSION_DEV=${BENTOV2_GATEWAY_VERSION}-dev
 BENTOV2_GATEWAY_CONTAINER_NAME=${BENTOV2_PREFIX}-gateway
 
@@ -77,7 +77,7 @@ BENTO_AUTH_DB_NETWORK="${BENTOV2_PREFIX}-auth-db-net"
 
 # - Authz service
 BENTO_AUTHZ_IMAGE=ghcr.io/bento-platform/bento_authorization_service
-BENTO_AUTHZ_VERSION=0.9.2
+BENTO_AUTHZ_VERSION=edge
 BENTO_AUTHZ_VERSION_DEV=${BENTO_AUTHZ_VERSION}-dev
 BENTO_AUTHZ_CONTAINER_NAME=${BENTOV2_PREFIX}-authz
 BENTO_AUTHZ_NETWORK=${BENTOV2_PREFIX}-authz-net
@@ -133,6 +133,8 @@ BENTOV2_SERVICE_REGISTRY_EXTERNAL_PORT=5010
 BENTOV2_SERVICE_REGISTRY_MEM_LIM=1G
 BENTOV2_SERVICE_REGISTRY_CPUS=1
 
+BENTO_SERVICE_REGISTRY_URL=${BENTOV2_PUBLIC_URL}/api/service-registry
+
 # Notification
 BENTOV2_NOTIFICATION_IMAGE=ghcr.io/bento-platform/bento_notification_service
 BENTOV2_NOTIFICATION_VERSION=3.1.4
@@ -249,6 +251,10 @@ BENTOV2_DRS_DEBUGGER_EXTERNAL_PORT=5682
 BENTOV2_DRS_MEM_LIM=2G
 BENTOV2_DRS_CPUS=2
 
+# Canonical/world-resolvable URL for DRS
+#  TODO: services should use the service registry instead
+BENTO_DRS_URL=${BENTOV2_PUBLIC_URL}/api/drs
+
 # Katsu-DB
 BENTOV2_KATSU_DB_IMAGE=postgres
 BENTOV2_KATSU_DB_VERSION=13
@@ -269,7 +275,7 @@ BENTOV2_KATSU_DB_CPUS=4
 
 # Katsu
 BENTOV2_KATSU_IMAGE=ghcr.io/bento-platform/katsu
-BENTOV2_KATSU_VERSION=8.0.1
+BENTOV2_KATSU_VERSION=edge
 BENTOV2_KATSU_VERSION_DEV=${BENTOV2_KATSU_VERSION}-dev
 BENTOV2_KATSU_CONTAINER_NAME=${BENTOV2_PREFIX}-katsu
 BENTO_KATSU_NETWORK=${BENTOV2_PREFIX}-katsu-net
@@ -290,6 +296,10 @@ BENTOV2_KATSU_CPUS=4
 # urls in templates.
 CHORD_METADATA_SUB_PATH=/api/metadata
 
+# Canonical/world-resolvable URL for Katsu
+#  TODO: services should use the service registry instead
+BENTO_KATSU_URL=${BENTOV2_PORTAL_PUBLIC_URL}${CHORD_METADATA_SUB_PATH}
+
 # Redis
 BENTOV2_REDIS_BASE_IMAGE=redis
 BENTOV2_REDIS_BASE_IMAGE_VERSION=7.0.15-alpine
@@ -336,6 +346,10 @@ BENTOV2_GOHAN_API_AUTHZ_ENABLED=false
 #BENTOV2_GOHAN_API_AUTHZ_AGREED_DISABLED_RISK=false
 BENTOV2_GOHAN_API_AUTHZ_REQHEADS=X-CUSTOM-1,X-CUSTOM-2
 
+# Canonical/world-resolvable URL for Gohan
+#  - TODO: services should use the service registry instead
+BENTO_GOHAN_URL=${BENTOV2_PORTAL_PUBLIC_URL}/api/gohan
+
 # -- Elasticsearch
 BENTOV2_GOHAN_ES_USERNAME=elastic
 # BENTOV2_GOHAN_ES_PASSWORD comes from default_config
@@ -382,7 +396,6 @@ BENTO_PUBLIC_EXTERNAL_PORT=8090
 BENTO_PUBLIC_DEBUG=false
 BENTO_PUBLIC_SERVICE_ID=${BENTOV2_PREFIX}-public
 BENTO_PUBLIC_CLIENT_NAME=BentoPublicDev
-BENTO_PUBLIC_KATSU_URL=http://${BENTOV2_KATSU_CONTAINER_NAME}:${BENTOV2_KATSU_INTERNAL_PORT}
 BENTO_PUBLIC_WES_URL=http://${BENTOV2_WES_CONTAINER_NAME}:${BENTOV2_WES_INTERNAL_PORT}
 BENTO_PUBLIC_GOHAN_URL=http://${BENTOV2_GOHAN_API_CONTAINER_NAME}:${BENTOV2_GOHAN_API_INTERNAL_PORT}
 BENTO_PUBLIC_PORTAL_URL=${BENTOV2_PORTAL_PUBLIC_URL}
@@ -392,7 +405,7 @@ BENTO_PUBLIC_PORTAL_URL=${BENTOV2_PORTAL_PUBLIC_URL}
 BENTO_BEACON_CONTAINER_NAME=${BENTOV2_PREFIX}-beacon
 BENTO_BEACON_NETWORK=${BENTOV2_PREFIX}-beacon-net
 BENTO_BEACON_IMAGE=ghcr.io/bento-platform/bento_beacon
-BENTO_BEACON_VERSION=0.15.2
+BENTO_BEACON_VERSION=edge
 BENTO_BEACON_VERSION_DEV=${BENTO_BEACON_VERSION}-dev
 BENTO_BEACON_INTERNAL_PORT=${BENTO_STD_SERVICE_INTERNAL_PORT}
 BENTO_BEACON_EXTERNAL_PORT=5000
@@ -402,10 +415,8 @@ BENTO_BEACON_MEM_LIM=2G
 BENTO_BEACON_CPUS=2
 BENTO_BEACON_CONFIG_DIR=${PWD}/lib/beacon/config
 
-BENTO_BEACON_GOHAN_BASE_URL=http://${BENTOV2_GOHAN_API_CONTAINER_NAME}:${BENTOV2_GOHAN_API_INTERNAL_PORT}
 BENTO_BEACON_KATSU_TIMEOUT=60
 BENTO_BEACON_GOHAN_TIMEOUT=60
-BENTO_BEACON_OIDC_ISSUER=${BENTOV2_AUTH_PUBLIC_URL}/auth/realms/${BENTOV2_AUTH_REALM}
 
 # cBioPortal
 

etc/bento_deploy.env

@@ -11,8 +11,8 @@ BENTO_GATEWAY_USE_TLS='true'
 BENTO_BEACON_ENABLED='false'  # Set to true if using Beacon!
 BENTO_BEACON_UI_ENABLED='false'
 BENTO_CBIOPORTAL_ENABLED='false'
-BENTO_MONITORING_ENABLED='false'
 BENTO_GOHAN_ENABLED='true'
+BENTO_MONITORING_ENABLED='false'
 
 #  - Switch to enable French translation in Bento Public
 BENTO_PUBLIC_TRANSLATED='true'
@@ -51,13 +51,17 @@ BENTOV2_AUTH_TEST_PASSWORD=
 BENTO_AUTH_DB_PASSWORD=  # TODO: SET ME WHEN DEPLOYING!
 BENTO_AUTHZ_DB_PASSWORD=  # TODO: SET ME WHEN DEPLOYING!
 
+#  - Aggregation/Beacon client ID/secret; client within BENTOV2_AUTH_REALM
+BENTO_AGGREGATION_CLIENT_ID=aggregation
+BENTO_AGGREGATION_CLIENT_SECRET=  # TODO: SET ME WHEN DEPLOYING!
+
 #  - WES Client ID/secret; client within BENTOV2_AUTH_REALM
 BENTO_WES_CLIENT_ID=wes
 BENTO_WES_CLIENT_SECRET=  # TODO: SET ME WHEN DEPLOYING!
 
 #  - Grafana Client ID/secret; client within BENTOV2_AUTH_REALM
 BENTO_GRAFANA_CLIENT_ID=grafana
-BENTO_GRAFANA_CLIENT_SECRET=
+BENTO_GRAFANA_CLIENT_SECRET=  # TODO: SET ME WHEN DEPLOYING IF GRAFANA IS ENABLED!
 # ---------------------------------------------------------------------
 
 BENTO_WEB_CUSTOM_HEADER=

etc/bento_dev.env

@@ -11,8 +11,8 @@ BENTO_GATEWAY_USE_TLS='true'
 BENTO_BEACON_ENABLED='true'
 BENTO_BEACON_UI_ENABLED='true'
 BENTO_CBIOPORTAL_ENABLED='false'
-BENTO_MONITORING_ENABLED='false'
 BENTO_GOHAN_ENABLED='true'
+BENTO_MONITORING_ENABLED='false'
 
 #  - Switch to enable French translation in Bento Public
 BENTO_PUBLIC_TRANSLATED='true'
@@ -51,6 +51,10 @@ BENTOV2_AUTH_ADMIN_PASSWORD=
 BENTOV2_AUTH_TEST_USER=
 BENTOV2_AUTH_TEST_PASSWORD=
 
+#  - Aggregation/Beacon client ID/secret; client within BENTOV2_AUTH_REALM
+BENTO_AGGREGATION_CLIENT_ID=aggregation
+BENTO_AGGREGATION_CLIENT_SECRET=
+
 #  - WES Client ID/secret; client within BENTOV2_AUTH_REALM
 BENTO_WES_CLIENT_ID=wes
 BENTO_WES_CLIENT_SECRET=

etc/default_config.env

@@ -17,6 +17,7 @@ BENTO_BEACON_ENABLED='true'
 BENTO_BEACON_UI_ENABLED='true'
 BENTO_CBIOPORTAL_ENABLED='false'
 BENTO_GOHAN_ENABLED='true'
+BENTO_MONITORING_ENABLED='false'
 
 #  - Switch to enable French translation in Bento Public
 BENTO_PUBLIC_TRANSLATED='true'
@@ -78,6 +79,9 @@ BENTOV2_AUTH_TEST_PASSWORD=
 #  - Auth (Keycloak) DB credentials
 BENTO_AUTH_DB_PASSWORD=
 BENTO_AUTHZ_DB_PASSWORD=
+#  - Aggregation/Beacon client ID/secret; secret to be filled by local.env - client within BENTOV2_AUTH_REALM
+BENTO_AGGREGATION_CLIENT_ID=aggregation
+BENTO_AGGREGATION_CLIENT_SECRET=
 #  - cBioPortal Client ID/secret; secret to be filled by local.env - client within BENTOV2_AUTH_REALM
 BENTO_CBIOPORTAL_CLIENT_ID=cbioportal
 BENTO_CBIOPORTAL_CLIENT_SECRET=

lib/aggregation/docker-compose.aggregation.yaml

@@ -8,8 +8,8 @@ services:
       - BENTO_DEBUG=False
       - USE_GOHAN=true
       - CORS_ORIGINS=${BENTO_CORS_ORIGINS}
-      - KATSU_URL=${BENTOV2_PORTAL_PUBLIC_URL}/api/metadata/
-      - SERVICE_REGISTRY_URL=${BENTOV2_PUBLIC_URL}/api/service-registry/
+      - KATSU_URL=${BENTO_KATSU_URL}/
+      - SERVICE_REGISTRY_URL=${BENTO_SERVICE_REGISTRY_URL}/
       - BENTO_AUTHZ_SERVICE_URL
     networks: 
       - aggregation-net

lib/beacon/docker-compose.beacon.yaml

@@ -5,26 +5,28 @@ services:
     container_name: ${BENTO_BEACON_CONTAINER_NAME}
     environment:
       - BENTO_UID
-      - GOHAN_BASE_URL=${BENTO_BEACON_GOHAN_BASE_URL}
+      - GOHAN_BASE_URL=${BENTO_GOHAN_URL}
       - KATSU_TIMEOUT=${BENTO_BEACON_KATSU_TIMEOUT}
-      - KATSU_BASE_URL=http://${BENTOV2_KATSU_CONTAINER_NAME}:${BENTOV2_KATSU_INTERNAL_PORT}
+      - KATSU_BASE_URL=${BENTO_KATSU_URL}
       - GOHAN_TIMEOUT=${BENTO_BEACON_GOHAN_TIMEOUT}
       - BENTO_BEACON_INTERNAL_PORT
       - INTERNAL_PORT=${BENTO_BEACON_INTERNAL_PORT}
       - BENTO_BEACON_EXTERNAL_PORT
       - BENTO_BEACON_DEBUGGER_INTERNAL_PORT
       - BENTO_BEACON_DEBUGGER_EXTERNAL_PORT
       - CONFIG_ABSOLUTE_PATH=/config/
-      - OIDC_ISSUER=${BENTO_BEACON_OIDC_ISSUER}
-      - CLIENT_ID=${BENTOV2_AUTH_CLIENT_ID}
       - BEACON_BASE_URL=${BENTOV2_PUBLIC_URL}/api/beacon
       - BENTO_BEACON_VERSION=${BENTO_BEACON_VERSION}
       - BENTO_PUBLIC_CLIENT_NAME
       - BENTOV2_DOMAIN
       - BENTOV2_PUBLIC_URL
       - BENTO_BEACON_UI_ENABLED
+      - DRS_URL=${BENTO_DRS_URL}
+      # Authorization
       - BENTO_AUTHZ_SERVICE_URL
-      - DRS_URL=${BENTOV2_PUBLIC_URL}/api/drs
+      - BENTO_OPENID_CONFIG_URL
+      - BEACON_CLIENT_ID=${BENTO_AGGREGATION_CLIENT_ID}
+      - BEACON_CLIENT_SECRET=${BENTO_AGGREGATION_CLIENT_SECRET}
     volumes:
       - ${BENTO_BEACON_CONFIG_DIR}:/config:ro
     networks: