Technical Specification: Translation of VCAP_SERVICES to Service Binding Files

[philippthun]

This document describes how service binding details, exposed to application processes in the VCAP_SERVICES environment variable, are translated into a list (tree) of files known as Service Binding Files.

Translation Process

1. The translation process begins with the system checking if the application process is enabled for file-based service bindings. If enabled, it builds a set of valid service binding files.

2. Each service binding's name is validated for uniqueness and compatibility. Duplicate or invalid binding names result in an IncompatibleBindings error. Binding names must match [a-z0-9\-.]{1,253} as specified in the Service Binding Specification for Kubernetes [1].

3. Multiple service binding files are generated for each service binding. The service binding's name acts as the directory name, while various properties of the service binding (such as credentials and attributes) act as file names.

4. The credentials attribute in VCAP_SERVICES is a JSON object. Each top-level key within this object will be matched to a file within the service binding's directory, and the corresponding value will be the file content. In case credentials contain a deeply nested structure, still only the top-level keys will be represented as files. Nested structures inside their value will be serialized as JSON objects. The same applies if the value is a list. (See example no. 1)

5. Existing credential keys can be overwritten by other attributes, that are VCAP_SERVICES attributes as defined in the Cloud Foundry Documentation [2] plus type and provider. The full list of reserved file names is: binding-guid, binding-name, instance-guid, instance-name, name, label, tags, plan, syslog-drain-url, volume-mounts, type, and provider. Overwriting an existing key does not result in an error. (See example no. 2)

6. All file names must match the same schema as the service binding's name. An invalid name results in an IncompatibleBindings error. The VCAP_SERVICES attributes are transformed to comply to this schema, i.e. underscores are replaced by hyphens, so that e.g. VCAP_SERVICES attribute binding_guid becomes file name binding-guid. (See example no. 3)

7. If a service binding value represents a list (like it is the case for tags and volume mounts), the file content will be a JSON representation of the array.

8. Empty lists or null values are omitted, i.e. no file will be created. (See example no. 3)

9. The total bytesize of the generated files is validated. If the total bytesize (of file paths and contents) exceeds the maximum allowed bytesize limit (default is 1,000,000 bytes), an IncompatibleBindings error is raised.

Examples

Example No. 1
-------------

VCAP_SERVICES= {
  "foo": [
    {
      "name": "foo",
      "credentials": {
        "simple": "value",
        "deeply": {
          "nested": "value"
        },
        "list": ["v", "a", "l", "u", "e"]
      }
    }
  ]
}

Service Binding Files:
  foo/name: foo
  foo/simple: value
  foo/deeply: {"nested":"value"}
  foo/list: ["v","a","l","u","e"]


Example No. 2
-------------

VCAP_SERVICES= {
  "foo": [
    {
      "name": "foo",
      "credentials": {
        "name": "user",
        "secret": "password"
      }
    }
  ]
}

Service Binding Files:
  foo/name: foo
  foo/secret: password


Example No. 3
-------------

VCAP_SERVICES= {
  "foo": [
    {
      "name": "foo",
      "binding_guid": "45436ca8-0a7c-45e3-9439-ca1b44db7a2b",
      "syslog_drain_url": null,
      "volume_mounts": []
    }
  ]
}

Service Binding Files:
  foo/name: foo
  foo/binding-guid: 45436ca8-0a7c-45e3-9439-ca1b44db7a2b

[1] https://servicebinding.io/spec/core/1.1.0/#workload-projection
[2] https://docs.cloudfoundry.org/devguide/deploy-apps/environment-variable.html#VCAP-SERVICES

[vlast3k]

@philippthun , we could have this case, where both properties are stored in the same way as files

VCAP_SERVICES= {
  "foo": [
    {
      "prop": "\[\"a\", \"a\"\]",
      "arr": ["a", "a"]
    }
  ]
}

foo/prop: ["a", "a"]
foo/arr: ["a", "a"]

Then w/o further metadata it is not possible to recover how the VCAP_SERVICES was intended to look like

[beyhan]

Hi @vlast3k,

this is specified in the RFC in https://github.com/cloudfoundry/community/blob/f8f28e9e8252e71770d8a762c4e78e60b321e6f0/toc/rfc/rfc-0030-add-support-for-file-based-service-binding.md?plain=1#L90-L92

[Samze]

If we have to do any drastic encoding based on the findings #4036 (comment) as part of this translation it would be good to know that existing libraries (like https://github.com/spring-cloud/spring-cloud-bindings) that automatically parse file based service bindings will function.

[beyhan]

Serialization

In our exploration of this topic, we've identified an open question: How should we write the binding content into files?

Given the service binding content (samples are based on experiments available in this repository):

{
    "name": "foo",
    "type": "db",
    "credentials": {
        "string_property": "[\"a\", \"a\"]",
        "array": ["a", "a"],
        "certificate": "-----BEGIN CERTIFICATE-----\nblaaaa\n-----END CERTIFICATE-----",
        "hash": {
            "a": "a"
        },
        "number": 1,
        "boolean": true,
        "password": "[\"dbsecret\"]"
    }
}

We've outlined two options below:

1. JSON Decoded:

./password
["dbsecret"]

./certificate
-----BEGIN CERTIFICATE-----
blaaaa
-----END CERTIFICATE-----

./hash
{"a":"a"}

./array
["a","a"]

./string_property
["a", "a"]

./number
1

./type
db

./json_content
{"a":"a"}

./boolean
true

./name
foo

2. JSON Encoded:

./password
"[\"dbsecret\"]"

./certificate
"-----BEGIN CERTIFICATE-----\nblaaaa\n-----END CERTIFICATE-----"

./hash
{"a":"a"}

./array
["a","a"]

./string_property
"[\"a\", \"a\"]"

./number
1

./type
"db"

./json_content
{"a":"a"}

./boolean
true

./name
"foo"

Based on the servicebindings.io specification and available libraries, the implicit assumption is that binding secret entries should contain plain strings. Libraries like spring-cloud-bindings read files as-is and trim them, making Option 1 more compatible to prevent issues with unintended quotes as shown here based on the samples above.

Backwards Compatibility

Choosing Option 1 (JSON Decoded) affects the translation from VCAP_SERVICES to servicebindings.io spec, as the format information might be lost. E.g. it is no possible any more to discover whether password or string_property from above was a string or JSON object. This will be an issue for generic libraries like java-cfenv and btp-environment-variable-access which are used to parse binding information in JSON format and provide typed property access. To address this challenge, you can consider two strategies:

1. Generate a .vcap_services file in the SERVICE_BINDING_ROOT folder (Example)

• Pros:

  • Ensures full backward compatibility
  • VCAP_SERVICES is a well-adopted API with documented format
  • Existing libraries will have minimal effort to adapt to this new file-based format
  • Doesn't violate the servicebindings.io spec, as files starting with . are ignored as shown here based on spring-cloud-bindings-client

• Cons:

  • The .vcap_services file will consume space within the 1 MB limit defined in RFC-0030.

2. Add a .metadata file for each service binding (Example)

• Pros:

  • Extends the servicebindings.io spec by adding valuable information lost during translation
  • Proven approach at SAP, as detailed in this blog post.
  • Doesn't violate the servicebindings.io spec, as files starting with . are ignored shown here based on spring-cloud-bindings-client

• Cons:

  • Not yet part of the servicebindings.io spec, but that could be changed
  • Full backward compatibility with the original VCAP_SERVICES format needs to be validated

[stephanme]

Another solution that avoids the drawbacks of the options 1 and 2 mentioned above is to go back to the original RFC0030 Add Support for File based Service Binding Information. We discussed here 2 alternatives:

1. The VCAP_SERVICES content is stored in a file which location is specified via the VCAP_SERVICES_FILE_PATH env var in the same format as the VCAP_SERVICES environment variable
2. Implement the K8s service binding specification.

We could actually implement both alternatives and give the CF user the choice by app feature flags (mutual exclusive), e.g.

• Option 3.1: file-based-service-bindings-vcap-services, file-based-vcap-services or file-based-service-bindings-cf
• Option 3.2: file-based-service-bindings-servicebinding-io, file-based-service-bindings or file-based-service-bindings-k8s - same as option 1 but w/o the .vcap_services file

(better naming proposals for the feature flags are welcome)

Given the following VCAP_SERVICES env var with the same service binding info as in the example above:

VCAP_SERVICES= {
  "db": [
    {
        "name": "foo",
        "label": "db",
        "credentials": {
            "string_property": "[\"a\", \"a\"]",
            "array": ["a", "a"],
            "certificate": "-----BEGIN CERTIFICATE-----\nblaaaa\n-----END CERTIFICATE-----",
            "hash": {
                "a": "a"
            },
            "number": 1,
            "boolean": true,
            "password": "[\"dbsecret\"]"
        }
    }
  ]
}

With enabled feature flag file-based-service-bindings-vcap-services, the following service bindings file is generated which contains the complete VCAP_SERVICES env var content (but not limited to 130k):

$VCAP_SERVICES_FILE_PATH
{
  "db": [
    {
        "name": "foo",
        "label": "db",
        "credentials": {
            "string_property": "[\"a\", \"a\"]",
            "array": ["a", "a"],
            "certificate": "-----BEGIN CERTIFICATE-----\nblaaaa\n-----END CERTIFICATE-----",
            "hash": {
                "a": "a"
            },
            "number": 1,
            "boolean": true,
            "password": "[\"dbsecret\"]"
        }
    }
  ]
}

• solves the 130k problem for VCAP_SERVICES env var
• makes migration for existing CF apps very easy
• doesn't lose any type information and therefore allows frameworks to interpret complex data correctly
• no information loss in case of conflicts between credential properties and well-known service binding properties like name, type, etc.

With enabled feature flag file-based-service-bindings-servicebinding-io, the following service binding files are generated according to the servicebinding.io spec in "JSON decoded" format and w/o any additional metadata:

$SERVICE_BINDING_ROOT/foo/password
["dbsecret"]

$SERVICE_BINDING_ROOT/foo/certificate
-----BEGIN CERTIFICATE-----
blaaaa
-----END CERTIFICATE-----

$SERVICE_BINDING_ROOT/foo/hash
{"a":"a"}

$SERVICE_BINDING_ROOT/foo/array
["a","a"]

$SERVICE_BINDING_ROOT/foo/string_property
["a", "a"]

$SERVICE_BINDING_ROOT/foo/number
1

$SERVICE_BINDING_ROOT/foo/json_content
{"a":"a"}

$SERVICE_BINDING_ROOT/foo/boolean
true

$SERVICE_BINDING_ROOT/foo/name
foo

$SERVICE_BINDING_ROOT/foo/type
db

$SERVICE_BINDING_ROOT/foo/provider
db

• servicebinding.io compliant, standard for k8s and CNBs
• compatible with existing libraries like spring-cloud-bindings
• type information gets lost but additional metadata can be added as the servicebinding.io spec evolves

Comparison of option 3 with the options 1 (Generate a .vcap_services file in the SERVICE_BINDING_ROOT folder) and 2 (Add a .metadata file for each service binding)

• Pros

  • user can chose between well-known VCAP_SERVICES format and servicebinding.io format depending on use case
  • no redundancy that consumes space within the 1MB limit
  • existing libraries can easily be adapted using file-based-service-bindings-vcap-services variant with full type information
  • servicebinding.io compatible option provided for existing CNB or k8s applications using file-based-service-bindings-servicebinding-io variant
  • avoids to specify now a non-standards metadata format as CF API which can't be changed later
  • file-based-service-bindings-servicebinding-io can easily be extended once servicebinding.io defines additional metadata
  • RFC-0030 is not blocked until servicebinding.io has defined additional metadata (in contrast to option 2)

• Cons

  • higher complexity for users as they have to chose from 2 file-based service binding variants
  • slightly higher implementation effort

[beyhan]

From the name options above I would vote for:

• file-based-vcap-services
• file-based-servicebinding-io (I don't see this as option above but I think it corresponds to file-based-vcap-services)

[dmikusa]

In regards to serialization, I don't see any of these approaches as being a problem, but 3.) would be my preference (@stephanme solution).

I think you give the app developer the choice on the format they want and it helps minimize change for them. If the app dev is using java-cfenv then they keep using vcap_services just from a file, which the library can handle. If they want to use spring cloud binding then it should be presented in service binding files.

Giving the choice means apps keep working and there's no messy migration.

Probably the only draw back I see there is for Buildpack authors. If you have Buildpacks that look at service bindings, like a number of Paketo Buildpacks, then they need to know if they should look at vcap_services or service bindings.

Fortunately in libcnb we have a layer that handles this. It is capable of presenting vcap_services data to buildpacks through the same interface they fetch service binding data. Presently it pulls from the env variable but we can easily look at a file instead.

[stephanme]

We discussed the proposals in the ARI WG meeting on 28.1. (unfortunately with low attendance).

If there are no strong arguments until end of this week we will go for option 3 (user can chose between a servicebinding.io and a vcap-services variant, both file based).

[Samze]

I would vote for option 3

[stephanme]

Decision: we continue with option 3.

[beyhan]

The RFC update is available in cloudfoundry/community#1069. Please put thumps up on it if it fine for you. FYI @Samze, @dmikusa