New RPC protocol proposal

[subpop]

This proposal is superseded by #93, however the proposal will remain here for posterity.

I'm planning on writing a new RPC protocol for workers to communicate with yggd. The current protocol has reached the limit of usefulness, and it's time to improve.

Removing Registration

I am deprecating the Registration method in favor of explicitly defined worker service files (see PR #37). These configuration files include everything necessary for a worker to register with the dispatcher, so an RPC method call is no longer necessary.

Adding a VARLINK interface

Currently, the only supported RPC framework is gRPC. This has proved very effective, but has led to some issues with Python workers. I've explored VARLINK some, and have had good results with some basic POC tests. I plan to support gRPC and VARLINK, and allow workers to select the protocol they use to communicate.

Proposal

Using the VARLINK notation, here's my current proposed protocol. A similar protocol interface could be written for gRPC.

com.redhat.yggdrasil.dispatch

These are methods implemented by yggd that workers can invoke.

# Interface to interact with yggd, the dispatch service.
interface com.redhat.yggdrasil.dispatch

# Relay sends data to the specified destination.
method Relay(destination: string, metadata: [string]string, data: object) -> (code: ?int, data: ?object)

# The destination could not be handled by the dispatch service.
error InvalidDestination ()

com.redhat.yggdrasil.worker

These are methods implemented by workers that yggd can invoke.

# Interface to interact with yggdrasil workers.
interface com.redhat.yggdrasil.worker

# Dispatch sends data to the worker along with the provided metadata.
method Dispatch(id: string, metadata: [string]string, data: object) -> (accepted: bool)

# The dispatched data was rejected by the worker.
error DispatchRejected (reason: string)

I'm sure this protocol is missing something. What is it leaving out? What other methods might we want to communicate between worker and dispatcher? I've deliberately not defined a Data type here with the idea of not exposing the concept of "messages" to workers. Good idea? Bad idea?

[eloycoto]

Hi,

Love the idea, thanks for sharing, comments:

Dispatch(worker->ygdd):

method Relay(destination: string, metadata: [string]string, data: object) -> (code: ?int, data: ?object)

Love this one, but:

• What is the destination? the URL to use? Where it should be defined? What kind of
  transport will be used, the default yggd one?
• I'll return a map with metadata. Like HTTP response headers, I
  can think about JWT.

On the dispatch, I would all the following methods:

method GetConfig()(config: [string]string)

This is to retrieve the config used by ygdd, we need to get access to the
Certs, so this is the best way.

method ReloadConfig() code: ?int

This is to reload config on ygdd, the main use case is when updating certs, and
be able to reload the TLSConfig. Have a working POC with something like this.

This is important for us.

Worker(ygdd->worker)

I miss why this can be used, I cannot see anything that can be used for us if
API responses are sent back. But I'm ok to have it, it's a good way to extend
something in future.

# Dispatch sends data to the worker along with the provided metadata.
method Dispatch(id: string, metadata: [string]string, data: object) -> (accepted: bool)

Maybe the GetConfig should dispatch an event that it's ConfigUpdated, and this
kind of things.

[subpop]

What is the destination? the URL to use? Where it should be defined? What kind of transport will be used, the default yggd one?

destination would vary, depending on the transport yggd was launched with. If it was launched with the HTTP transport, then the destination value should be a valid HTTP URL that yggd would perform an HTTP POST request to, using metadata as X- headers and data as the request body. If yggd was launched with the MQTT transport, destination would be a "directive" value; the cloud application to which the message is destined. This doesn't handle the case of the detachedContent worker yet either; where the "cloud-to-client" data comes over MQTT, but the "client-to-cloud" content goes back over HTTP.

I'll return a map with metadata. Like HTTP response headers, I
can think about JWT.

Yes, metadata is a (in Go type parlance) a map[string]string]. It is entirely worker defined. yggd doesn't care what the contents of the map are. I was thinking that in the case of the HTTP transport, these map to X- headers, but if you're thinking about putting JWT values in, you're assuming you can use headers like Authentication? I believe the HTTP transport does not maintain Authentication in its HTTP client, but it seems like something that should be provided by yggd and not something that should be client managed.

[subpop]

method GetConfig()(config: [string]string)

This makes a lot of sense. I can include some sort of ability for workers to get access to yggd configuration values.

method ReloadConfig() code: ?int

This is interesting. yggd doesn't assume it needs to reload the configuration. It just reads it once at startup and currently watches a few special files for changes. I think it might make sense for yggd to watch its configuration file and, if configured, its certificates for modify events; and reload them when received. This would eliminate the need for workers to have to track when the files change and manually tell yggd to reload.

[subpop]

I miss why this can be used, I cannot see anything that can be used for us if API responses are sent back. But I'm ok to have it, it's a good way to extend something in future.

This is how yggd tells a worker that it has a message. If your worker doesn't ever expect to receive a message from an external source, then yes, you would not use this method. Other workers do use this to initiate work, however.

[ShimShtein]

I see this design removes the registration method from the protocol.
For some features we were planning to use this method to pass metadata from the worker to message emitter.

How such call would be implemented using the new protocol?
Would it be modeled as a Relay call to a well known destination?

[subpop]

I see this design removes the registration method from the protocol. For some features we were planning to use this method to pass metadata from the worker to message emitter. How such call would be implemented using the new protocol? Would it be modeled as a Relay call to a well known destination?

Yes. I'm planning to remove the dynamic registration in favor of static worker activation. See #37 for details on this change. The metadata that workers passed in the registration will need to be split into two forms: (1) static metadata values will get defined in the worker configuration file (this works for things like worker version strings and stuff that doesn't change). (2) dynamic metadata is basically just data being sent from a worker, so I think it would need to be handled as a call to Relay. You could use either metadata or data for the values; by either mapping them into the metadata map, or serializing them into bytes in the data parameter. Does this seem like it would work for you?

[ShimShtein]

Well, in this case it would be either Relay or direct API call. Sounds good.

[pkliczewski]

@subpop Do we have any time line for this change?

[jakub-dzon]

Using the current protocol we've implemented de-registration/graceful decommission operation in the worker API (#41 and #54) and I think that would make sense to have something similar in the new API also as a separate operation.
@subpop - WDYT?

[subpop]

@subpop Do we have any time line for this change?

No, no timeline right now, sorry.

[subpop]

Using the current protocol we've implemented de-registration/graceful decommission operation in the worker API (#41 and #54) and I think that would make sense to have something similar in the new API also as a separate operation. @subpop - WDYT?

It sounds like there's going to be some sort of Event dispatch system that we'll need here. Workers want to know about things happening in the Dispatcher. Workers getting notified of a disconnection, a reconnection, a configuration change, etc. If the RPC protocol doesn't offer native events, we can always fake it by adding methods on the worker interface that yggd calls. I'll draft some changes to the interfaces above that include events like these.

[subpop]

I recently picked this work back up and while working on it, I came to the conclusion that VARLINK does not bring much additional benefit over gRPC to the protocol. If anything, it's actually less featured. While I was thinking over what protocols do bring additional functionality, I decided to look into using D-Bus as a worker/dispatcher IPC protocol. This proved to be very effective, and brings a number of nice features to the protocol:

Signals

We added a rudimentary "event" API in #54, but D-Bus has first-class support for signals. This means we can define in the protocol specification, signals like:

<signal name="Event">
    <arg type="u" name="name" />
</signal>

Emitting this signal is as simple as:

conn.Emit("/com/redhat/yggdrasil/Dispatcher1", "com.redhat.yggdrasil.Dispatcher1.Event", DispatcherEventReceivedDisconnect)

A client (i.e. worker) can register with the bus to receive notifications when these signals are emitted:

conn.AddMatchSignal(dbus.WithMatchObjectPath("/com/redhat/yggrasil/Dispatcher1", dbus.WithMatchInterface("com.redhat.yggdrasil.Dispatcher1"), dbus.WithMatchMember("Event")

Properties

D-Bus has first-class support for object properties. Properties can be connected to in order to receive notifications when values change. I haven't added them yet, but with Properties, we'll be able to have DefaultConfig values exported onto the Dispatcher1 object on the bus for workers to receive or get when they want to explore the current runtime configuration of the Dispatcher. I also plan on adding a Features property to the com.redhat.yggdrasil.Worker1 interface that the Dispatcher1 will subscribe to. If a worker wants to trigger a republish of the connection-status message, with an updated feature map, it can change its Features and a subscription handler in the dispatcher will pick up that change and republish the connection-status.

Overall, I'm very pleased with the way the D-Bus implementation has turned out. I published a branch that includes a working implementation of a D-Bus Dispatcher service and an echo worker. Please take a look and let me know if you see any glaring omissions or oversights.

One other thing to note here is that for the sake of simplicity, I dropped gRPC from the codebase with this branch. While supporting two protocols is technically possible, it added a fair amount of complexity for little benefit.

[subpop]

I put some more thought into this and wrote a follow-up proposal that describes my proposal in better detail. #93