Skip to content

Commit

Permalink
feat: add standalone STS component (#355)
Browse files Browse the repository at this point in the history
* add standalone STS runtime (wip)

* fix IntelliJ deployment

* updated README

* terraform fmt

* format readme toc

* updated README

* fix ci
  • Loading branch information
paullatzelsperger authored Oct 4, 2024
1 parent 4ffd586 commit dc5c951
Show file tree
Hide file tree
Showing 15 changed files with 428 additions and 49 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/run-e2e-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ jobs:
cluster_name: dcp-demo

- name: "Load runtime images into KinD"
run: kind load docker-image controlplane:latest dataplane:latest identity-hub:latest catalog-server:latest -n dcp-demo
run: kind load docker-image controlplane:latest dataplane:latest identity-hub:latest catalog-server:latest sts:latest -n dcp-demo

- name: "Install nginx ingress controller"
run: |-
Expand Down
4 changes: 2 additions & 2 deletions .run/remote/IdentityHub PROVIDER.run.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@
<option name="SERVER_MODE" value="false" />
<option name="SHMEM_ADDRESS" />
<option name="HOST" value="localhost" />
<option name="PORT" value="1045" />
<option name="PORT" value="1044" />
<option name="AUTO_RESTART" value="false" />
<RunnerSettings RunnerId="Debug">
<option name="DEBUG_PORT" value="1045" />
<option name="DEBUG_PORT" value="1044" />
<option name="LOCAL" value="false" />
</RunnerSettings>
<method v="2" />
Expand Down
43 changes: 37 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
* [5.1 Build the runtime images](#51-build-the-runtime-images)
* [5.2 Create the K8S cluster](#52-create-the-k8s-cluster)
* [5.3 Seed the dataspace](#53-seed-the-dataspace)
* [5.4 Debugging MVD in Kubernetes](#54-debugging-mvd-in-kubernetes)
* [6. Differences between Kubernetes and IntelliJ](#6-differences-between-kubernetes-and-intellij)
* [6.1 In-memory databases](#61-in-memory-databases)
* [6.2 Memory-based secret vaults](#62-memory-based-secret-vaults)
Expand Down Expand Up @@ -276,10 +277,10 @@ documentation for more information.

```shell
# Create the cluster
kind create cluster -n dcp-demo --config deployment/kind.config.yaml
kind create cluster -n mvd --config deployment/kind.config.yaml

# Load docker images into KinD
kind load docker-image controlplane:latest dataplane:latest identity-hub:latest catalog-server:latest -n dcp-demo
kind load docker-image controlplane:latest dataplane:latest identity-hub:latest catalog-server:latest sts:latest -n mvd

# Deploy an NGINX ingress
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml
Expand Down Expand Up @@ -331,7 +332,7 @@ Once all the deployments are up-and-running, the seed script needs to be execute
output similar to this:

```shell
./seed-k8s.sh
./seed-k8s.sh


Seed data to "provider-qna" and "provider-manufacturing"
Expand All @@ -358,6 +359,35 @@ _the `node` warnings are harmless and can be ignored_
> Failing to run the seed script will leave the dataspace in an uninitialized state and cause all connector-to-connector
> communication to fail.
### 5.4 Debugging MVD in Kubernetes

All of MVD's runtime images come with remote JVM debugging enabled by default. This is configured by setting an
environment variable

```
JAVA_TOOL_OPTIONS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=<DEBUG_PORT>"
```

All runtimes use port **1044** for debugging, unless configured otherwise in terraform. The only thing left to do for
you is to create a Kubernetes port-forwarding:

```shell
kubectl port-forward -n mvd service/consumer-controlplane 1044:1044
```

This assumes the default Kubernetes namespace `mvd`. Note that the port-forward targets a `service` to have it
consistent across pod restarts, but targeting a specific pod is also possible. Please refer to the official
documentation for details.

The host port (the value after the `:`) is completely arbitrary, and should be altered if multiple runtimes are debugged
in parallel.

When creating a "Remote JVM Debug" run configuration in IntelliJ it is important to select the appropriate module
classpath. Those are generally located in the `launchers/` directory.

Please also refer to the [official IntelliJ tutorial](https://www.jetbrains.com/help/idea/tutorial-remote-debug.html) on
how to do remote debugging.

## 6. Differences between Kubernetes and IntelliJ

The focus with the Kubernetes deployment is to achieve a "one-click-deployment" (don't count them, it's more than 1)
Expand All @@ -384,12 +414,13 @@ all secrets that need to be accessed by multiple components must be pre-populate

### 6.3 Embedded vs Remote STS

While in the Kubernetes deployment the SecureTokenService is embedded into the IdentityHub runtime, in the IntelliJ
While in the Kubernetes deployment the SecureTokenService (S)S is a stand-alone component, in the IntelliJ
deployment it is embedded into the controlplane. The reason for this is, that during seeding a participant context and
an STS Account is created. This includes a (generated) client secret, that gets stored in the vault.

In the IntelliJ case that vault is isolated in IdentityHub, with no way to access it from the connector's controlplane.
This makes it necessary that the STS be embedded in the controlplane directly.
In the IntelliJ case that vault is purely in-memory and is isolated in IdentityHub, with no way to access it from the
connector's controlplane. So the connector's controlplane and IdentityHub physically cannot share any secrets. To
overcome this, STS is simply embedded in the controlplane directly.

In the Kubernetes deployment this limitation goes away, because a dedicated vault service (HashiCorp Vault) is used,
which is accessible from either component.
Expand Down
19 changes: 17 additions & 2 deletions deployment/consumer.tf
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ module "consumer-connector" {
}
vault-url = "http://consumer-vault:8200"
namespace = kubernetes_namespace.ns.metadata.0.name
sts-token-url = module.consumer-identityhub.sts-token-url
sts-token-url = "${module.consumer-sts.sts-token-url}/token"
}

# consumer identity hub
Expand All @@ -43,7 +43,22 @@ module "consumer-identityhub" {
password = "consumer"
url = "jdbc:postgresql://${module.consumer-postgres.database-url}/consumer"
}
namespace = kubernetes_namespace.ns.metadata.0.name
namespace = kubernetes_namespace.ns.metadata.0.name
sts-accounts-api-url = module.consumer-sts.sts-accounts-url
}

# consumer standalone STS
module "consumer-sts" {
depends_on = [module.consumer-vault]
source = "./modules/sts"
humanReadableName = "consumer-sts"
namespace = kubernetes_namespace.ns.metadata.0.name
database = {
user = "consumer"
password = "consumer"
url = "jdbc:postgresql://${module.consumer-postgres.database-url}/consumer"
}
vault-url = "http://consumer-vault:8200"
}

# consumer vault
Expand Down
52 changes: 27 additions & 25 deletions deployment/modules/identity-hub/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -136,31 +136,33 @@ resource "kubernetes_config_map" "identityhub-config" {

data = {
# IdentityHub variables
EDC_API_AUTH_KEY = "password"
EDC_IH_IAM_ID = var.participantId
EDC_IAM_DID_WEB_USE_HTTPS = false
EDC_IH_IAM_PUBLICKEY_ALIAS = local.public-key-alias
EDC_IH_API_SUPERUSER_KEY = var.ih_superuser_apikey
WEB_HTTP_PORT = var.ports.web
WEB_HTTP_PATH = "/api"
WEB_HTTP_IDENTITY_PORT = var.ports.ih-identity-api
WEB_HTTP_IDENTITY_PATH = "/api/identity"
WEB_HTTP_PRESENTATION_PORT = var.ports.presentation-api
WEB_HTTP_PRESENTATION_PATH = "/api/presentation"
WEB_HTTP_DID_PORT = var.ports.ih-did
WEB_HTTP_DID_PATH = "/"
WEB_HTTP_STS_PORT = var.ports.sts-api
WEB_HTTP_STS_PATH = "/api/sts"
JAVA_TOOL_OPTIONS = "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=${var.ports.ih-debug}"
EDC_IAM_STS_PRIVATEKEY_ALIAS = var.aliases.sts-private-key
EDC_IAM_STS_PUBLICKEY_ID = var.aliases.sts-public-key-id
EDC_MVD_CREDENTIALS_PATH = "/etc/credentials/"
EDC_VAULT_HASHICORP_URL = var.vault-url
EDC_VAULT_HASHICORP_TOKEN = var.vault-token
EDC_DATASOURCE_DEFAULT_URL = var.database.url
EDC_DATASOURCE_DEFAULT_USER = var.database.user
EDC_DATASOURCE_DEFAULT_PASSWORD = var.database.password
EDC_SQL_SCHEMA_AUTOCREATE = true
EDC_API_AUTH_KEY = "password"
EDC_IH_IAM_ID = var.participantId
EDC_IAM_DID_WEB_USE_HTTPS = false
EDC_IH_IAM_PUBLICKEY_ALIAS = local.public-key-alias
EDC_IH_API_SUPERUSER_KEY = var.ih_superuser_apikey
WEB_HTTP_PORT = var.ports.web
WEB_HTTP_PATH = "/api"
WEB_HTTP_IDENTITY_PORT = var.ports.ih-identity-api
WEB_HTTP_IDENTITY_PATH = "/api/identity"
WEB_HTTP_PRESENTATION_PORT = var.ports.presentation-api
WEB_HTTP_PRESENTATION_PATH = "/api/presentation"
WEB_HTTP_DID_PORT = var.ports.ih-did
WEB_HTTP_DID_PATH = "/"
WEB_HTTP_STS_PORT = var.ports.sts-api
WEB_HTTP_STS_PATH = "/api/sts"
JAVA_TOOL_OPTIONS = "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=${var.ports.ih-debug}"
EDC_IAM_STS_PRIVATEKEY_ALIAS = var.aliases.sts-private-key
EDC_IAM_STS_PUBLICKEY_ID = var.aliases.sts-public-key-id
EDC_MVD_CREDENTIALS_PATH = "/etc/credentials/"
EDC_VAULT_HASHICORP_URL = var.vault-url
EDC_VAULT_HASHICORP_TOKEN = var.vault-token
EDC_DATASOURCE_DEFAULT_URL = var.database.url
EDC_DATASOURCE_DEFAULT_USER = var.database.user
EDC_DATASOURCE_DEFAULT_PASSWORD = var.database.password
EDC_SQL_SCHEMA_AUTOCREATE = true
EDC_STS_ACCOUNT_API_URL = var.sts-accounts-api-url
EDC_STS_ACCOUNTS_API_AUTH_HEADER_VALUE = "password"
}
}

Expand Down
12 changes: 6 additions & 6 deletions deployment/modules/identity-hub/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ variable "ports" {
default = {
web = 7080
debug = 1044
ih-debug = 1045
ih-debug = 1044
ih-did = 7083
ih-identity-api = 7081
presentation-api = 7082
Expand All @@ -61,11 +61,6 @@ variable "credentials-dir" {
description = "JSON object containing the credentials to seed, sorted by human-readable participant name"
}

variable "participant-list-file" {
type = string
default = "./assets/participants/participants.k8s.json"
}

variable "ih_superuser_apikey" {
default = "c3VwZXItdXNlcg==.c3VwZXItc2VjcmV0LWtleQo="
description = "Management API Key for the Super-User. Defaults to 'base64(super-user).base64(super-secret-key)"
Expand Down Expand Up @@ -105,4 +100,9 @@ variable "database" {
user = string
password = string
})
}

variable "sts-accounts-api-url" {
description = "Base URL for the STS Accounts API"
type = string
}
150 changes: 150 additions & 0 deletions deployment/modules/sts/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
#
# Copyright (c) 2024 Metaform Systems, Inc.
#
# This program and the accompanying materials are made available under the
# terms of the Apache License, Version 2.0 which is available at
# https://www.apache.org/licenses/LICENSE-2.0
#
# SPDX-License-Identifier: Apache-2.0
#
# Contributors:
# Metaform Systems, Inc. - initial API and implementation
#

resource "kubernetes_deployment" "sts" {
metadata {
name = var.humanReadableName
namespace = var.namespace
labels = {
App = var.humanReadableName
}
}

spec {
replicas = 1
selector {
match_labels = {
App = var.humanReadableName
}
}
template {
metadata {
labels = {
App = var.humanReadableName
}
}
spec {
container {
image = "sts:latest"
name = "sts"
image_pull_policy = "Never"

env_from {
config_map_ref {
name = kubernetes_config_map.sts-config.metadata[0].name
}
}
port {
container_port = var.ports.accounts
name = "accounts-port"
}

port {
container_port = var.ports.sts
name = "sts-port"
}

# Uncomment this to assign (more) resources
# resources {
# limits = {
# cpu = "2"
# memory = "512Mi"
# }
# requests = {
# cpu = "250m"
# memory = "50Mi"
# }
# }

liveness_probe {
http_get {
path = "/internal/check/liveness"
port = var.ports.web
}
failure_threshold = 10
period_seconds = 5
timeout_seconds = 30
}

readiness_probe {
http_get {
path = "/internal/check/readiness"
port = var.ports.web
}
failure_threshold = 10
period_seconds = 5
timeout_seconds = 30
}

startup_probe {
http_get {
path = "/internal/check/startup"
port = var.ports.web
}
failure_threshold = 10
period_seconds = 5
timeout_seconds = 30
}
}
}
}
}
}

resource "kubernetes_config_map" "sts-config" {
metadata {
name = "${var.humanReadableName}-config"
namespace = var.namespace
}

## Create databases for keycloak and MIW, create users and assign privileges
data = {
JAVA_TOOL_OPTIONS = "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=${var.ports.debug}"
WEB_HTTP_ACCOUNTS_PORT = var.ports.accounts
WEB_HTTP_ACCOUNTS_PATH = var.accounts-path
WEB_HTTP_PORT = var.ports.web
WEB_HTTP_PATH = "/internal"
WEB_HTTP_STS_PORT = var.ports.sts
WEB_HTTP_STS_PATH = var.sts-path
EDC_API_ACCOUNTS_KEY = "password"
EDC_DATASOURCE_DEFAULT_URL = var.database.url
EDC_DATASOURCE_DEFAULT_USER = var.database.user
EDC_DATASOURCE_DEFAULT_PASSWORD = var.database.password
EDC_SQL_SCHEMA_AUTOCREATE = true

EDC_VAULT_HASHICORP_URL = var.vault-url
EDC_VAULT_HASHICORP_TOKEN = var.vault-token
}
}

resource "kubernetes_service" "sts-service" {
metadata {
name = "${var.humanReadableName}-service"
namespace = var.namespace
}
spec {
selector = {
App = kubernetes_deployment.sts.spec.0.template.0.metadata[0].labels.App
}
port {
name = "accounts-port"
port = var.ports.accounts
target_port = var.ports.accounts
}
port {
name = "sts-port"
port = var.ports.sts
target_port = var.ports.sts
}
}
}
Loading

0 comments on commit dc5c951

Please sign in to comment.