NFR - Protect and encrypt sensitive data when stored persistently with strict data separation

[DanielaWuensch]

Feature Request

Non-functional Requirement:
Protect and encrypt sensitive data when stored persistently (with strict data separation between the sensitive data and non-sensitive data and data from different customers)
As a company, which operates the EDC I want to ensure that sensitive data is stored securely (encrypted). Currently the persistence is used in the data management api and the data is not encrypted.
Highest priority would have the asset->dataAddress as it contains credentials of the backend service. These credentials should be protected with VH priority. Credentials should not be stored insecure and perfect place for it would be the vault.
Extract the the dataAddress in data mgt api and store it in the vault (which ensures secure persistence)
In addition all other data persisted should be encrypted with a key provided by the admin.
encrypt the db data

Which Areas Would Be Affected?

all, including DPF, CI, build, transfer, etc._

Why Is the Feature Desired?

Security Requirement

Solution Proposal

sensitive data need alway to be stored securely (encrypted).

Type of Issue

non-functional requirement

Checklist

all sensitive data are stored securely (encrypted).

[jimmarino]

The data address does not contain any secrets (this appears to be a misinterpretation of the code). All secrets are, by EDC design, stored in the Vault.

[lholthof]

If I get you right, then your suggestion would lead to a two step asset creation.
Create vault secrets plus use the corresponding vault-keys in POST /asset.

I also do not see an issue in having a vault key in the data address. I just wanted to use it by convention to keep the convenience as-is for the data mgt API users, to have one call creating an asset including all details of the dataAddress and not to care about vault-keys.
With the prefixing the API user would have the choice to define which properties should be put into the vault. At the same time this leave a point of failure. Therefore why not combine both? Typed properties will be put into the vault and in addition the ones prefixed with secret:. Some will consider the URL as sensitive data and some might not agree to that. Like this everyone has the choice while we will ensure that e.g. an authCode (typed as Secret) will always be put into the vault.

Maybe one improvement on using the vault-key by convention (based on asset-id) would be to hash the content and use this hash as vault-key. The vault-key would be an additional property in the dataAddress. With this repeated same secrets will not consume an individual vault entry all the time. At the same time the items in the vault cache will be less which improves performance.

[jimmarino]

I think we are introducing too many topics into the discussion at once and this is leading to confusion. So far, we have one actionable issue:

1. Creating a way (preferably typed) to ensure secret values are not passed in HttpDataAddress (other DataAddress usages correctly do this)

Performance was also mentioned but I would suggest putting that aside for the purposes of this discussion since it is a broader issue.

Another point that is raised relates to the creation of an Asset and its associated DataAddress (this is distinct from the actionable issue since data addresses are used throughout the codebase in diverse contexts for dereferencing assets). The Vault secret should be generated out-of-band and stored in the Vault when the asset contents are stored at the location referenced by the data address (which is not necessarily when the Asset and DataAddress are created in the EDC). Users entering assets should not have access to those secrets nor should they populate them in the Vault. This is a separate topic, though.

[dileep-ramachandrarao]

Following the thread above, I have few clarifications and for the purpose of better understanding, I would like to discuss using a simple example.

Currently, during Asset creation if the data endpoint is password protected then the data address would look like this:

"dataAddress": {
        "properties": {
            "type": "HttpData",
            "endpoint": "http://provider-backend-service:9191/data/{{digitalTwinSubmodelId}}",
	    "authCode": "Basic an7uthklGywTMnMkM="
        }
    }

Here authCode is a sensitive data, which contains Base64 encoded username and password and it is persisted without encryption.

As suggested in the above thread, prefix the sensitive properties using secret: and store them in the vault.
By following further discussion, I see that we could either

Option 1: hash all the sensitive properties and store them in the vault like:

vaultContent={"secret:endpoint": "http://provider-backend-service:9191/data/{{digitalTwinSubmodelId}}",
"secret:authCode": "Basic an7uthklGywTMnMkM="}
hash(vaultContent)= h849dkckkd //would be the vault key

Within the vault, the sensitive properties could be accessed using the vault key:

VaultKey: h849dkckkd
VaultContent: {"secret:endpoint": "http://provider-backend-service:9191/data/{{digitalTwinSubmodelId}}",
"secret:authCode": "Basic an7uthklGywTMnMkM="}

Following would be stored in the DB:

"dataAddress": {
    "properties": {
      "secret:endpoint": "h849dkckkd",
      "type": "HttpData",
      "secret:authCode": "h849dkckkd"
    }
  }

OR

Option 2: hash of the respective sensitive property:

vaultContent={"secret:endpoint": "http://provider-backend-service:9191/data/{{digitalTwinSubmodelId}}"}
hash(vaultContent)= 123564987
VaultKey: 123564987

vaultContent={"secret:authCode": "Basic an7uthklGywTMnMkM=}
hash(vaultContent)= abcdklskdjf
VaultKey: abcdklskdjf

Following would be stored in the DB:

"dataAddress": {
    "properties": {
      "secret:endpoint": "123564987",
      "type": "HttpData",
      "secret:authCode": "abcdklskdjf"
    }
  }

Does either of the approaches a right way to address this issue?

[dileep-ramachandrarao]

Trying to revive this conversation. @jimmarino @ndr-brt any opinion on this?

[jimmarino]

Putting sensitive data in the DataAddress is incorrect. If code is doing this, it needs to be fixed. The only data that should be placed in the data address is non-sensitive data and pointers to where sensitive data may be retrieved from a vault.

There is no need to hash or encrypt sensitive data in a vault since the vault implementation handles that.

HTH.