Merge branch 'releases/v17' into add-beacon-network-config

README.md

@@ -44,6 +44,7 @@ that make up the Bento platform.
 
 ### Migration documents
 
+* [v16 to v17](./docs/migrating_to_17.md)
 * [v15.2 to v16](./docs/migrating_to_16.md)
 * [v15.1 to v15.2](./docs/migrating_to_15_2.md)
 * [v15 to v15.1](./docs/migrating_to_15_1.md)

docker-compose.dev.yaml

@@ -65,6 +65,11 @@ services:
           - ${BENTOV2_DOMAIN}
           - ${BENTOV2_PORTAL_DOMAIN}
           - ${BENTOV2_AUTH_DOMAIN}
+      monitoring-net:
+          aliases:
+          - ${BENTOV2_DOMAIN}
+          - ${BENTOV2_PORTAL_DOMAIN}
+          - ${BENTOV2_AUTH_DOMAIN}
       public-net:
         aliases:
           - ${BENTOV2_DOMAIN}
@@ -232,3 +237,14 @@ services:
   cbioportal:
     ports:
       - "${BENTO_CBIOPORTAL_EXTERNAL_PORT}:${BENTO_CBIOPORTAL_INTERNAL_PORT}"
+
+  grafana:
+    ports:
+      - "3000:3000"
+    environment:
+    # Workaround for self signed certificates in dev
+      - GF_AUTH_GENERIC_OAUTH_TLS_SKIP_VERIFY_INSECURE=true
+
+  loki:
+    ports:
+      - "3100:3100"

docker-compose.yaml

@@ -13,6 +13,7 @@ include:
       - lib/event-relay/docker-compose.event-relay.yaml
       - lib/gohan/docker-compose.gohan.yaml  # Optional feature; controlled by a compose profile
       - lib/katsu/docker-compose.katsu.yaml
+      - lib/logs/docker-compose.logs.yaml
       - lib/notification/docker-compose.notification.yaml
       - lib/public/docker-compose.public.yaml  # Optional feature; controlled by a compose profile
       - lib/redis/docker-compose.redis.yaml

docs/deployment.md

@@ -99,3 +99,11 @@ ls certs/gateway/letsencrypt/live/
 ```
 
 If all went well, the `old-bento.example.com` domain should be redirected to `bento.example.com` in a browser.
+
+## Discovery configuration
+
+Bento can serve censored data publicly if configured to do so. This allows anonymous users to take a glimpse into the
+data hosted by a Bento node.
+
+When deploying a Bento instance, make sure that the discovery settings are configured properly at the necessary levels.
+Consult the [public discovery](./public_discovery.md) documentation for more details.

docs/installation.md

@@ -284,8 +284,10 @@ specified in the step above.
 ./bentoctl.bash init-auth
 ```
 
-**If using an external identity provider**, only start the cluster's gateway
-after setting `CLIENT_SECRET` in your local environment file:
+After running `init-auth`, be sure to put all client secrets into your `local.env` file!
+
+**If using an external identity provider**, only start the cluster's gateway after setting various `*_CLIENT_SECRET` 
+variables in your local environment file:
 
 ```bash
 ./bentoctl.bash run gateway
@@ -305,7 +307,7 @@ utilize new variables generated during the OIDC configuration.
 
 ## 6. Configure permissions
 
-### a. Create superuser permissions in the new Bento authorization service
+### a. Create superuser permissions in the Bento authorization service
 
 First, run the authorization service and then open a shell into the container:
 
@@ -325,24 +327,54 @@ which in Keycloak should be a UUID.
 
 ### b. Create grants for the Workflow Execution Service (WES) OAuth2 client
 
-Run the following commands to set up authorization for the WES client. Don't forget to replace `ISSUER_HERE` by the 
-issuer URL!
+Run the following commands to set up authorization for the WES client. 
+**Don't forget to replace `<ISSUER_HERE>` with the issuer URL!**
 
 ```bash
 # This grant is a temporary hack to get permissions working for v12/v13. In the future, it should be removed.
 bento_authz create grant \
-  '{"iss": "ISSUER_HERE", "client": "wes"}' \
+  '{"iss": "<ISSUER_HERE>", "client": "wes"}' \
   '{"everything": true}' \
   'view:private_portal'
 
 # This grant gives permission to access and ingest data into all projects and the reference genome service
 bento_authz create grant \
-  '{"iss": "ISSUER_HERE", "client": "wes"}' \
+  '{"iss": "<ISSUER_HERE>", "client": "wes"}' \
   '{"everything": true}' \
   'query:data' 'ingest:data' 'ingest:reference_material' 'delete:reference_material'
 ```
 
-### c. *Optional step:* Assign portal access to all users in the instance realm
+### c. Create a grant for the aggregation and Beacon services
+
+Run the following commands to set up authorization for the aggregation/Beacon client. 
+**Don't forget to replace `<ISSUER_HERE>` with the issuer URL!**
+
+```bash
+# In the future, view:private_portal will need to be removed from this grant.
+bento_authz create grant \
+  '{"iss": "<ISSUER_HERE>", "client": "aggregation"}' \
+  '{"everything": true}' \
+  'query:data' 'view:private_portal'
+```
+
+
+### d. Configure public data access for all users, including anonymous visitors (if desired):
+
+To configure public data access, run the following command in the authorization service container. Note that with the
+`full` value, **THIS GIVES FULL DATA ACCESS TO EVERYONE WHO VISITS YOUR INSTANCE!**
+
+```bash
+# Configure public data access
+# ----------------------------
+# The level below ("counts") preserves previous functionality. Other possible options are:
+#  - none - will do nothing.
+#  - bool - for censored true/false discovery, but in effect right now forbids access.
+#  - counts - for censored count discovery.
+#  - full - allows full data access (record-level, including sensitive data such as IDs), uncensored counts, etc.
+bento_authz public-data-access counts
+```
+
+### e. Assign portal access to all users in the instance realm
 
 We added a special permission, `view:private_portal`, to Bento v12/v13 in order to carry forward the current
 'legacy' authorization behaviour for one more major version. This permission currently behaves as a super-permission,

docs/migrating_to_17.md

@@ -0,0 +1,91 @@
+# Migrating to Bento v17
+
+Key points:
+
+* Bento now has observability tools to help monitor the services (Grafana). Some setup is required for this feature to 
+  work.
+* Katsu discovery endpoints now have an authorization layer. 
+  * Data that used to be completely public by default (i.e., 
+    censored counts) now requires a permission (`query:project_level_counts` and/or `query:dataset_level_counts`), and 
+    thus a grant in the authorization service. 
+  * Beacon now requires a client ID/secret and an authorization service grant to access uncensored data.
+* Katsu discovery is now more granular, and can be configured to the project or dataset level, in addition to the 
+  instance level. See the [Public data discovery configuration](./public_discovery.md) document for more information.
+* ...
+
+
+## 1. Stop Bento
+
+```bash
+./bentoctl.bash stop
+```
+
+
+## 2. Update images
+
+```bash
+./bentoctl.bash pull
+```
+
+
+## 3. Set up credentials for aggregation/Beacon and, optionally, set up Grafana
+
+If you wish to enable Grafana, you first must enable the monitoring feature in your `local.env` file:
+
+```bash
+BENTO_MONITORING_ENABLED='true'
+```
+
+To create the client secrets for aggregation/Beacon and Grafana (if the latter is enabled), run the following commands:
+
+```bash
+./bentoctl.bash start auth
+./bentoctl.bash init-auth
+```
+
+**Reminder:** Make sure to put the client secret(s) generated by `init-auth` into your `local.env` file!
+
+Aggregation/Beacon data access authorization will not work until an authorization service grant is configured; 
+see step 4 below.
+
+
+## 4. Set up aggregation/Beacon permissions and public data access grants
+
+Now that Beacon uses a client ID/secret to get authorized, uncensored data access for discovery, a grant must be 
+configured to give the aggregation/Beacon client data access.
+
+Another change to permissions: starting from Bento v17, anonymous visitors do not have access to see censored counts 
+data by default, even if a discovery configuration has been set up. For anonymous visitors to access data, a level 
+(`bool`, `counts`, `full`) must be chosen and passed to the `bento_authz` CLI command below.
+
+```bash
+./bentoctl.bash shell authz
+
+# Configure aggregation/Beacon permissions
+# ----------------------------------------
+# This assumes the aggregation/Beacon client ID is "aggregation". 
+# <ISSUER_HERE> MUST be replaced with your actual issuer value.
+#  - The query:data permission gives access to Katsu endpoints which are properly authz-enabled.
+#  - The view:private_portal permission gives access to Katsu and Gohan endpoints where the proxy still manages access.
+#    This permission will be removed in an uncoming version.
+bento_authz create grant \
+  '{"iss": "<ISSUER_HERE>", "client": "aggregation"}' \
+  '{"everything": true}' \
+  'query:data' 'view:private_portal'
+
+# Configure public data access
+# ----------------------------
+# The level below ("counts") preserves previous functionality. Other possible options are:
+#  - none - will do nothing.
+#  - bool - for censored true/false discovery, but in effect right now forbids access.
+#  - counts - for censored count discovery.
+#  - full - allows full data access (record-level, including sensitive data such as IDs), uncensored counts, etc.
+bento_authz public-data-access counts
+```
+
+
+## 5. Start Bento
+
+```bash
+./bentoctl.bash start
+```

docs/public_discovery.md

@@ -0,0 +1,80 @@
+# Public data discovery configuration
+
+New in Bento v17.
+
+Previously, the public data configuration given to Katsu (`lib/katsu/config.json`) was applied on all the metadata 
+contained in the service. This configuration declares which fields can be queried publicly for discovery purposes, 
+which charts to display and which censorship rules to apply on the results.
+
+Katsu can hold multiple projects/datasets that may use different fields, require specific charts or custom
+`extra_properties` schemas at the project level.
+Therefore, there is a need to tailor the discovery configuration at different levels to properly represent the 
+particularities in the information of a project or dataset.
+
+Bento v17 gives the ability to specify a scoped Discovery configuration at the following levels:
+-   Dataset
+    -   Optional at dataset creation
+    -   For scoped queries on public endpoints targeting a project and dataset:
+        -   Katsu will use the dataset's discovery configuration, if one exists.
+        -   If no configuration is found, falls back to the parent project's discovery.
+-   Project
+    -   Optional at project creation
+    -   For scoped queries on public endpoints targeting a project only:
+        -   Katsu will use the project's discovery configuration, if one exists.
+        -   If no configuration is found, falls back to the node's config.
+-   Node
+    -   Optional during Katsu deployment
+    -   Uses the legacy `lib/katsu/config.json` file mount
+    -   For non-scoped queries on public endpoints:
+        -   Katsu will use the node's discovery, if one exists.
+        -   If no node configuration is found, Katsu will respond with a 404 status.
+    -   For scoped queries on public endpoints:
+        -   Katsu will fallback on the node's discovery if the project and/or dataset in the scope don't have one
+        -   If no node configuration is found, Katsu will respond with a 404 status.
+
+
+## Creating a discovery configuration file.
+
+Follow these steps in order to add public discovery for a given scope.
+
+1. Decide at which level the discovery configuration will be applied.
+2. Create a copy of `etc/katsu.config.example.json`, use it as a base template
+3. Modify the `fields` section so that it includes the fields of interest.
+4. Modify the `search` section, include the fields from the previous section to make them searchable
+5. Modify the `overview` section in order to specify which fields should be rendered as charts and how.
+6. Set the desired censorship rules in the `rules` section.
+
+The resulting JSON file needs to respect the JSON-schema defined in Katsu [here](https://github.com/bento-platform/katsu/blob/develop/chord_metadata_service/discovery/schemas.py).
+
+## Using a discovery configuration file.
+
+With the valid discovery configuration file in hand, you can now add it to Katsu.
+
+### Apply the discovery configuration at the node level:
+
+This operation must be done by a Bento node administrator.
+```bash
+# Move the file to Katsu's config volume
+cp <discovery config JSON file> ./lib/katsu/config.json
+
+# restart Katsu to load the config.
+./bentoctl.bash restart katsu
+```
+
+### Apply the discovery configuration at the project or dataset level
+
+These operations must be performed in `bento_web` by an authenticated user 
+with the `edit:dataset`, `create:dataset`, `edit:project`, `create:project` permissions.
+
+Before creating/editing a project/dataset to specify a discovery configuration, 
+make sure that the JSON file is in Bento's dropbox.
+
+At project/dataset creation, leaving the discovery field empty is permitted.
+Specify a discovery config by selecting the desired file from the drop-down options.
+
+![Specifying a discovery at project creation.](./img/discovery_proj_creation.png)
+
+During project/dataset editing, three radio buttons are shown, allowing the user to pick the existing value,
+a new value from file, or none. The 'none' option is equivalent to leaving the field empty at creation.
+
+![Modifying the discovery when editing a project](./img/discovery_proj_edit.png)