Merge pull request #607 from OpenSourceBrain/release/0.5.4

Release/0.5.4

.dockerignore

@@ -0,0 +1,5 @@
+/.overrides
+/deployment
+.gitignore
+/.vscode
+.pytest_cache

.gitignore

@@ -7,3 +7,4 @@ skaffold.yaml
 /deployment.yaml
 .pytest_cache
 /.overrides
+.hypothesis

LICENSE

@@ -0,0 +1,27 @@
+The MIT License (MIT)
+
+Copyright © 2022 Silver Lab, University College London
+http://silverlab.org/
+
+All rights reserved. This program and the accompanying materials
+are made available under the terms of the MIT License
+which accompanies this distribution, and is available at
+http://opensource.org/licenses/MIT
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE
+USE OR OTHER DEALINGS IN THE SOFTWARE.

README.md

@@ -7,12 +7,12 @@ An updated version of the Open Source Brain platform
 ### Prerequisites
 
 The OSB deployment is built on top of [CloudHarness](https://github.com/MetaCell/cloud-harness).
-The deployment process is based on Python 3.7+ scripts. It is recommended to setup a virtual 
+The deployment process is based on Python 3.7+ scripts. It is recommended to setup a virtual
 environment first.
 
-With conda: 
+With conda:
 ```bash
-conda create --name osb python=3.7
+conda create --name osb python=3.9
 conda activate osb
 ```
 
@@ -40,7 +40,7 @@ CloudHarness scripts script automate the deployment process.
 To manually create the helm chart to use on any Kubernetes deployment, run:
 
 ```
-harness-deployment cloud-harness . 
+harness-deployment cloud-harness .
 ```
 ### Cluster setup
 
@@ -108,9 +108,18 @@ Connect your docker registry with minikube with:
 
 Then run:
 ```
-harness-deployment cloud-harness . -l  -n osblocal -d osb.local -u -dtls -m build -e local -i osb-portal 
+harness-deployment cloud-harness . -l  -n osblocal -d osb.local -u -dtls -e local -i osb-portal
 ```
 You do not need to run the port-forwarding commands on the local deployment.
+
+Here, you can modify the argument of the `-e` option to select what environment you want to deploy.
+These correspond to the files from the `osb-portal/deploy` directory.
+So, selecting the environment will load specific overriding configuration files (like `[APP_NAME]/deploy/values-[ENV].yaml`) specific to the environment.
+
+If you only want to run the back-end in the minikube deployment, change the `osb-portal` to `workspaces`.
+You can then use `npm start:minikube` to point `npm` to the local minikube back-end.
+Note that the domain in `package.json` for the `start:minikube` command should match the namespace used for minikube.
+
 Finally, run skaffold to build and run the images on minikube:
 
 ```
@@ -123,7 +132,7 @@ On making local changes, you can re-run the `harness-deployment` command to upda
 
 With the registry on localhost:5000 run:
 ```
-harness-deployment cloud-harness . -l  -n osblocal -d osb.local -u -dtls -m build -e local -i osb-portal -r registry:5000
+harness-deployment cloud-harness . -l  -n osblocal -d osb.local -u -dtls -e local -i osb-portal -r registry:5000
 ```
 
 See below to learn how to configure Minikube and forward the registry.

applications/accounts-api/api/openapi.yaml

@@ -42,6 +42,33 @@ paths:
       operationId: ready
       summary: Test if application is ready to take requests
   /users:
+    get:
+      summary: Get list of users based on query
+      operationId: get_users
+      parameters:
+      - description: query filter
+        in: query
+        name: query_string
+        required: false
+        schema:
+          type: string
+      security:
+      - bearerAuth: []
+      responses:
+        "200":
+          content:
+            application/json:
+              schema:
+                type: object
+                properties:
+                  users:
+                    type: array
+                    items:
+                      $ref: '#/components/schemas/User'
+          description: Get list of users
+      tags:
+      - users
+      x-openapi-router-controller: accounts_api.controllers.users_controller
     post:
       requestBody:
         content:
@@ -169,4 +196,4 @@ tags:
 - name: infrastructure
   description: ""
 - name: users
-  description: ""
+  description: ""

applications/accounts-api/backend/accounts_api/controllers/users_controller.py

@@ -38,6 +38,18 @@ def get_user(userid):  # noqa: E501
         return "User not found", 404
 
 
+def get_users(query={}):
+    """get all users
+
+    :param query: user filter
+    :type query: str
+
+    :rtype: {}
+    """
+
+    return {'users': user_service.get_users(query)}
+
+
 def update_user(userid, user=None):  # noqa: E501
     """get_user
 

applications/accounts-api/backend/accounts_api/openapi/openapi.yaml

@@ -46,6 +46,33 @@ paths:
       - infrastructure
       x-openapi-router-controller: accounts_api.controllers.infrastructure_controller
   /users:
+    get:
+      summary: Get list of users based on query
+      operationId: get_users
+      parameters:
+      - description: query filter
+        in: query
+        name: query_string
+        required: false
+        schema:
+          type: string
+      security:
+      - bearerAuth: []
+      responses:
+        "200":
+          content:
+            application/json:
+              schema:
+                type: object
+                properties:
+                  users:
+                    type: array
+                    items:
+                      $ref: '#/components/schemas/User'
+          description: Get list of users
+      tags:
+      - users
+      x-openapi-router-controller: accounts_api.controllers.users_controller
     post:
       operationId: create_user
       requestBody:

applications/accounts-api/backend/accounts_api/services/user_service.py

@@ -3,6 +3,7 @@
 from accounts_api.models import User
 from cloudharness.auth import AuthClient
 from cloudharness import log
+import typing
 # from cloudharness.models import User as CHUser # Cloudharness 2.0.0
 
 class UserNotFound(Exception): pass
@@ -33,8 +34,23 @@ def get_user(userid: str) -> User:
     return user
 
 
+def get_users(query: str) -> typing.List[User]:
+    try:
+        client = AuthClient()
+        kc_users = client.get_users(query)
+    except KeycloakError as e:
+        raise Exception("Unhandled Keycloak exception") from e
+    all_users = []
+    for kc_user in kc_users:
+        auser = map_user(kc_user)
+        auser.email = None  # strip out the e-mail address
+        all_users.append(auser)
+
+    return all_users
+
+
 def map_user(kc_user) -> User:
-    user = User.from_dict(kc_user)
+    user = kc_user if isinstance(kc_user, dict) else User.from_dict(kc_user._raw_dict)
     if 'attributes' not in kc_user or not kc_user['attributes']:
         kc_user['attributes'] = {}
 
@@ -57,7 +73,7 @@ def map_user(kc_user) -> User:
     return user
 
 
-def update_user(userid, user: User):
+def update_user(userid, user: User):    
     client = AuthClient()
 
     try:
@@ -69,15 +85,15 @@ def update_user(userid, user: User):
             'firstName': user.first_name or current_user['firstName'],
             'lastName': user.last_name or current_user['lastName'],
             'attributes': {
-                **current_user.get('attributes', {}),
+                **(current_user.get('attributes') or {}),
                 **({('profile--' + k): user.profiles[k] for k in user.profiles} if user.profiles else {}),
                 'avatar': user.avatar,
                 'website': user.website
             }
         }
 
         admin_client.update_user(userid,  updated_user)
-        return map_user({**current_user, **updated_user})
+        return get_user(userid)
     except KeycloakError as e:
         if e.response_code == 404:
             raise UserNotFound(userid)

applications/accounts/Dockerfile

@@ -1,9 +1,12 @@
-FROM quay.io/keycloak/keycloak:11.0.3
+FROM quay.io/keycloak/keycloak:16.1.0
 
 # add kubectl
 USER root
 COPY --chmod=0755 scripts/create_api_user.sh /opt/jboss/startup-scripts/create_api_user.sh
 USER jboss
 
 # Customize keycloak look
-COPY themes/custom /opt/jboss/keycloak/themes/custom
+COPY themes/custom /opt/jboss/keycloak/themes/custom
+
+# keycloak kafka listener plugin
+COPY plugins/metacell-admin-event-listener-bundle-1.0.0.ear /opt/jboss/keycloak/standalone/deployments/

applications/accounts/deploy/resources/realm.json

@@ -1,3 +1,31 @@
+
+{{- define "deploy_accounts_utils.role" }}
+{
+  "id": {{ uuidv4 | quote }},
+  "name": {{ .role| quote }},
+  "composite": false,
+  "clientRole": true,
+  "containerId": {{ .app.harness.name | quote }},
+  "attributes": {}
+}
+{{- end}}
+{{- define "deploy_accounts_utils.user" }}
+    {
+        "username": {{ .user.username | quote }},
+        "email": {{ .user.email | default .user.username | quote }},
+        "enabled": true,
+        "credentials": [
+            {
+                "type": "password",
+                "value": {{ .user.passwword | default "test" | quote }}
+            }
+        ],
+        "realmRoles": {{ .user.realmRoles | toJson }},
+        "clientRoles": {
+            {{ .app.harness.name | quote }}: {{ .user.clientRoles | toJson }}
+        } 
+    }
+    {{- end}}
 {
         "id": {{ .Values.namespace | quote }},
         "realm": {{ .Values.namespace | quote }},
@@ -15,6 +43,15 @@
         "duplicateEmailsAllowed": false,
         "resetPasswordAllowed": true,
         "editUsernameAllowed": true,
+        "users": [
+            {{- range $app := .Values.apps }}
+              {{- if (hasKey $app.harness "accounts")  }}
+                {{- range $i, $user := $app.harness.accounts.users }}{{if $i}},{{end}}
+                {{ include "deploy_accounts_utils.user" (dict "root" $ "app" $app "user" $user) }}
+                {{- end }}
+              {{- end }}
+            {{- end }}
+        ],
         "roles": {
             "realm": [
                 {
@@ -43,7 +80,19 @@
                     "containerId": {{ .Values.namespace | quote }},
                     "attributes": {}
                 }
-            ]
+            ],
+            "client": {
+            {{- range $app := .Values.apps }}
+              {{- if (hasKey $app.harness "accounts")  }}
+                {{ $app.harness.name | quote }}: [
+                {{- range $i, $role := $app.harness.accounts.roles }}
+                {{if $i}},{{end}}
+                {{ include "deploy_accounts_utils.role" (dict "root" $ "app" $app "role" $role) }}
+                {{- end }}
+                ]
+              {{- end }}
+            {{- end }}
+            }
         },
         "clients": [
             {
@@ -81,6 +130,44 @@
                     "microprofile-jwt"
                 ]
             },
+            {{- range $app := .Values.apps }}
+            {
+                "id": {{ $app.harness.name | quote }},
+                "clientId": {{ $app.harness.name | quote }},
+                "name": {{ $app.harness.name | quote }},
+                "surrogateAuthRequired": false,
+                "enabled": true,
+                "alwaysDisplayInConsole": false,
+                "clientAuthenticatorType": "client-secret",
+                "secret": {{ randAlphaNum 20 | quote }},
+                "redirectUris": [
+                  "*"
+                ],
+                "webOrigins": [
+                  "*",
+                  "+"
+                ],
+                "notBefore": 0,
+                "bearerOnly": false,
+                "consentRequired": false,
+                "standardFlowEnabled": true,
+                "implicitFlowEnabled": false,
+                "directAccessGrantsEnabled": true,
+                "serviceAccountsEnabled": false,
+                "publicClient": true,
+                "frontchannelLogout": false,
+                "protocol": "openid-connect",
+                "attributes": {
+                  "backchannel.logout.session.required": "true",
+                  "backchannel.logout.revoke.offline.tokens": "false"
+                },
+                "authenticationFlowBindingOverrides": {},
+                "fullScopeAllowed": true,
+                "nodeReRegistrationTimeout": -1,
+                "defaultClientScopes": [],
+                "optionalClientScopes": []
+              },
+              {{- end }}
             {
                 "id": "111caf43-3d26-484d-8dc9-7fa911ac221c",
                 "clientId": {{ .Values.apps.accounts.webclient.id | quote }},

applications/accounts/themes/custom/login/login-verify-email.ftl

@@ -9,5 +9,6 @@
         <p class="instruction">
             ${msg("emailVerifyInstruction2")} <a href="${url.loginAction}">${msg("doClickHere")}</a> ${msg("emailVerifyInstruction3")}
         </p>
+        <p>If you are still having problems with this process, please get in contact at <a href="mailto:[email protected]">[email protected]</a></p>
     </#if>
 </@layout.registrationLayout>