How to get HTTP Push to Work?

[efiege]

We are trying to get the http-push, outlined in in the data-plane-transfer-client diagram, to work. When the consumer starts the transfer-process, a token should be generated, which can later be used by the provider at arbitrary times to push data to the consumer.

So the process is as follows:

• before 0.: The consumer prepares a token which can be used by the provider to push data
• 0.: The consumer sends a DataRequest, containing the token in the additional properties, to it's own Control Plane
• 2.: The DataRequest is sent to the providers Control Plane, which uses the data-plane-transfer-client extension to execute the request using a remote Data Plane
• 3.: A DataFlowRequest is sent from the RemoteDataPlaneTransferClient to the DataPlaneControlApiController.
  • The Controller uses the data-plane-framework to start the transfer-process
  • The PipelineService uses Source and SinkFactories corresponding to the asset's type and the requested dataDestination. This could be the factories provided by the data-plane-http extension
• 4.: The asset's http endpoint is hit by the Data Plane. Using the data-plane-http extension currently there is no possibility to transfer the token from the sourceDataAddress to the destinationDataAddress
• 5.: The provider pushes it's data using the provided token. This can be done invoking the providers http endpoint directly or through the providers data plane (5'.).

At the moment we see no possibility to transfer the generated token to the providers asset endpoint without writing additional code at step 4.

Do we have to implement a custom ResourceManifestGenerator as stated at #1386?

Is it possible to transfer the data over the data plane of the provider (step 5') right now?

Thanks for your help!

[jimmarino]

As a first step, you need to decide when the secret key is created on the consumer, for example, whether is it done prior to the data request in step 0 or is done as part of the provision step on the consumer.

Once that is decided, the secret must be passed as part of the data request (but not part of the data address). On the provider, the secret will be stored in the vault using the key name associated with the data address. At that point, the component responsible for accessing the data on the provider side will need to receive the data address and use the key name to pull the secret from the vault and push data to the consumer.

Does that make sense?

[efiege]

Totally makes sense, nice solution! We will try to get that working.

Thanks for the fast reply!

[efiege]

You mentioned that

the component responsible for accessing the data on the provider side will need to receive the data address

We currently see three possibilities transferring the data address to the component:

1. Control Plane: During provisioning on the providers side
2. Control Plane: Implementing a DataFlowController, which uses a OkHttpClient to Post the data address to the component
3. Data Plane: Using the data-plane-transfer-client, where the DataSource will push the data address to the component and the DataSink will ignore the data

Do you have a recommendation which option to choose?

Option 3 could look like:

[jimmarino]

I think what you are proposing is more complicated than it needs to be. In the modified diagram, the DPF only serves as a layer of unnecessary indirection to notify an external system to send data to an endpoint.

To get started simplifying this architecture, I would step back and define the requirements of the data transfer more clearly:

1. Is it finite or non-finite (e.g., events, stream) data?
2. If it is non-finite, how are notifications of new events, data, etc., made?
3. What protocol and transport layer is required between the client and provider?
4. What delivery semantics are required?

For example, suppose the data transfer is finite and involves sending a simple payload over HTTPS from an external system on the provider side. In that case, all that is needed is a DataFlowController. There is no need for a DPF. In this case, the DataAddress will be passed to the flow controller.

If, on the other hand, the data is an ongoing stream of events, you may want to use the DPF. In this case, the data source would be a DPF extension that "subscribes" to receive events. When an event is received, it passes the event into the DPF TransferService, using an input stream data source, which will then send it down a pipeline to a sink responsible for dispatching the event to the remote endpoint. The important thing here is the DPF is performing the transfer. Otherwise, it makes no sense to use the DPF (implement a DataFlowController or the data plane interface yourselves).

HTH