Synapse performance & scaling

[gerbyzation]

Hi!

We're exploring the workflow engine space and came across serverless workflow spec and synapse, which has piqued our interest.

For our intended usecase performance and scalability is crucial, so looking to get a better understanding of what performance and scalability synapse can offer/is designed for. From conversation on slack with @cdavernas I believe it hasn't been tested at scale yet.

My main two questions are:

• What level of scale has synapse been designed for?
• To what level of scale has synapse been tested so far?

We might be able to help with some testing if needed.

Thanks!
Gerrit

[cdavernas]

Hello @gerbyzation!

First of all, sorry for the delay of my response, we got our hands full with a demo we showcased at the last ServerlessWorkflow workshop.

What level of scale has synapse been designed for?

Synapse strives to run workflow at scale and in perfect isolation in a cloud native environment.

The current model introduces a Single Point of Failure, as the Synapse server acts as a centralized workflow domain/data-plane controller. This can - and will - however be mitigated with some kind of clustering mechanics (Raft, for one) that will be introduced no sooner than 0.5 to ensure full horizontal scalability (and real-time consistency) of said SPOFs. Your suggestions, uses-cases, help are by the way more than welcome on the subject!!!

Note: The first release of Synapse, in turn, used to leverage ETCD to manage workflow-related data in a decentralized manner, but restricted possible deployment environments (basically: kube only). Plus, some might consider that a malpractice: using CRDs for managing data-plane concepts.

[cdavernas]

Yes! After long reflections and several tests projects, we decided to head for a Resource Oriented Architecture (ROA) approach, similar - but not limited to - what Kubernetes does with ETCD. As a matter of consequence, we have started Hylo, which is an ROA API to proxy DB providers, such as Mongo, ETCD (via K8s API), Redis, File System, etc.

Using that approach, we no longer impose our understanding of how workflows should be stored, validated and executed, while offering implementers to add their own interpreter or business logic using Validating/Mutating webhooks as well as generic 'WorkflowAgents'.

In other words, workflow definitions and their instances are stored (but no longer managed by ) the Synapse server, which now offers several features to observe, act and react upon resource-related events, provoked by independant, autonomous agents, possibly interpreting other DSLs than ServerlessWorkflow.

The release candidate offers a default, fully customizable version of a SW workflow agent implementation, running workflows using the process-oriented approach of the current version.

Thanks to the revamp, Synapse gains the following advantages;

• Isolation: all components of the solution are now fully independant of each other, at the cost of them being all tightly coupled to the backing database of the ROA API.
• Scalability: thanks to component isolation, the solution is fully scalable, both horizontally and vertically. The choke point has been migrated to the ROA API backing database
• No SPOF: thanks to component isolation, the solution can survive to most catastrophic failure scenarii, as long as the ROA API backing database is up and running, which is a non-concern given the fact the later is (or should be) distributed and highly available
• Lean data: as we learned painfully with the actual release, storing the workflow data along with its context was a very bad idea: in some cases, said data was so large it was taking up to minutes to simply list workflow instances. We decided to segregate the workflow data into a dedicated "Document" database (MinIO, for instance), referenced in context objects (such as WorkflowInstance) using simple URIs. This guarantees that, no matter the size of the data manipulated by workflows, intertacting with resources such as WorkflowInstances is be near instantaneous.
• Polyglot: because of component isolation, and thanks to the Agent pattern that we are using, Agents can be developped in any and all languages, as long as they can interact with the (HTTP REST) ROA API backing database. Furthermore, the revamp introduces DSLs and DSL Extensions, which allow implementers to develop agents against basically any kind of language, while retaining management features offered by Synapse.

[ElanHasson]

Sounds cool. If i understand correctly, you've essentially shifted everything to CRDs and an operator-type model?

If so, quite a shift :)

How does this change impact synapse? Is the data still event sourced?

[gerbyzation]

This all does sound quite interesting! Is there any more documentation on this design?

[cdavernas]

Sounds cool. If i understand correctly, you've essentially shifted everything to CRDs and an operator-type model?

@ElanHasson You understood perfectly indeed, that's a fine way to sumarize it!

If so, quite a shift :)

Yes, and a painful one too. But I felt back then that the architecture put in place was not agile enough, and was having an obvious applicative bottleneck on an easily overstrained API server, which I was hosting along with the Runtime API (big mistake).

How does this change impact synapse?

It impacts Synapse in the way that all domain objects such as Workflows, WorkflowInstances, etc. are now declarative K8s-like Custom Resources, enforcing schema but not any underlying business logic, which delegated to operators. The management API is now hosted on an optional API server app, which is needed by the Dashboard. Executing workflows is delegated to Agent, which in turn starts a WorkflowRunner, which executes a single instance.

Is the data still event sourced?

No, it is not. But it could be added if needed, requiring an Event-Sourcing implementation of Hylo's IRepository interface.

I thought it was more important to put emphasis on reading/consuming speed than having granular, overloaded (with workflow data) domain events, which were anyways described somewhat differently by workflow activities.

Is it something that sounds reasonable to you? I'm very happy to have the opportunity to read/hear your take on it all.

[cdavernas]

This all does sound quite interesting! Is there any more documentation on this design?

Here are some of my personal diagrams, if that helps:

[cdavernas]

To what level of scale has synapse been tested so far?

I think that the maximum amount of (small, simple) workflow instances that we have been running concurrently is about a hundred, without noticeable impact on overall performances.
The UI, at the time, was howevever no able to handle hundreds of events concurrently, and won't ever be able to, I suppose, without throttling.

[gerbyzation]

Have you pushed this any further in the meantime? What we're looking at will start around the thousands and needs a clear path of scaling towards hundred of thousands/millions if adoption lives up to it.

[cdavernas]

We have not yet pushed it further than that, no. I guess that if you have the infrastructure, it is something that you could easily test, no? If so, please do share your findings!!!

Server wise, I think that is should be ok, Kestrel being able to handle up to 7 MILLION requests per second.

Application wise, well, that's a different story. The design of the actual data models, which include workflow data, and produce domain and integration events that carry that workflow data over. In specific workflows, defining a foreach state for example, workflow data can quickly become very big, affecting negatively both commands and queries. This is aggravated by deserialization/serialization mechanisms, especially with GRPC (proto serialization of large JSON payloads).

That being said, the processing work is essentially made on the worker, which runs on a container or a pod. So, aside from possible service slow down due to large workflow data, if your infrastructure can hold it, so should Synapse.

As said to @ElanHasson, an overhaul is being worked on that address distribution concerns, and offers additionality granularity with independent API Server, Workflow Agents and Workflow Runner applications.

[cdavernas]

As for proper unit and/or integration testing, all the help is more than welcome!

Concerning the scenarii, what do you have in mind? Maybe starting by producing something like a classic 10/100/1000 test case?

Concerning the tech-side of tests, there are a ton of test frameworks for NET CORE, but I personnaly often rely on XUnit, along with FluentAssertions to write them (a little bit more) painlessly. But anything would do just fine IMHO.

[cdavernas]

@ElanHasson @gerbyzation If you guys are still interested, please check out the long overdue v1.0.0-*. Aside from all changes announced above, it implements the latest version of the Serverless Workflow DSL, which has also been refactored from top to bottom!

I'm eager to hear your feedback!

[ElanHasson]

Hey @cdavernas,

Been a while!

I've seen the intense fury of issue alerts come out of the project!

I admit, I haven't been keeping up. I'll take a look and let you know any thoughts I have.