Add partition key, content-type to EventHub

[acco]

ContentType

Right now, we send messages to Event Hub in a JSON body like this:

[
  { "Body": "<some base64-encoded body>" },
  { "Body": "<some base64-encoded body>" },
]

However, it looks like you can set a content type for messages. When set to JSON, we don't have to base64 encode the body. Messages look nicer in the Azure console/Data Explorer:

Can we set this on individual messages in the batch request?

If not, can we set it in the non-batch request? If so, we should just do it and set batch_size=1 to start.

Partition key

Setting the partition key is important for ordering. This is how SQS derives a group ID (in its pipeline):

consumer
|> Consumers.group_column_values(record_or_event_data)
|> Enum.join(",")

Looking at the Event Hub Go SDK, I'm pretty sure we can add a partition key to individual messages:

https://github.com/Azure/azure-event-hubs-go/blob/master/event.go#L37-L48

However, it's not clear where to put that information.

We should intercept requests to the Azure API to figure out what the payload needs to look like to do this. To intercept, we can either:

• Change the base URL of the Go SDK to POST a batch of messages to a local webserver
• Use a proxy (Charles) to capture the request to Azure

[acco]

@yordis Writing the docs for Event Hubs, these two felt relevant. If we can get partition key relatively cheaply (<1h of investigation), which I think we can do by introspecting one of the SDKs, we should do it.

May need to introspect in order to capture both of these properties.

[acco]

Argh, just noticed the Go SDK uses AMQP, not REST - so this may not actually be possible so long as we're using the REST API.

We can send individual events with a partition ID, but that will mean needing to manage partition count on our side etc.

Alternatively, we can move over EventHubs to using either our Kafka or AMQP adapters, which will give us more bells and whistles.

Moving to backlog, but we'll look at this again by request.

[yordis]

@acco could we do the same as the RabbitMQ then?