XWorker: An SDK to work with async workers

[ajatprabha]

Open Questions

• Should we make use of generics for Job payload and Handler signatures?

[sonnes]

Check out https://github.com/riverqueue/river

i really liked the ease of use of their APIs

[sonnes]

I’m trying to use it for kafka publishing #47

[sonnes]

I have been thinking about this for a while. These are few things that I am concerned about:

1. gocraft/work is a pretty outdated library. It is not maintained anymore. I know we maintain a fork of it, but it is just band-aids. gocraft's primary constraint is only works with Redis.

2. Choosing a async worker framework/library is a big decision. Primarily because every framework has different trade-offs, UI, and APIs. Bending these frameworks to work with our SDK significantly increases the complexity of the SDK. Once a project chooses a framework, it is unlikely to switch to another one or use multiple frameworks.

Alternatively, theses are some options we can consider:

1. Adopt a new library like https://github.com/riverqueue/river. It is Postgres based and should easily integrate with most of our projects. It is already structured as a SDK with pluggable drivers.

2. We can build our own async worker framework. The generics based SDK of River can be a good starting point. This only makes sense if we really need to support different databases - Redis, Badger, etc AND a small subset of River's features.

[ajatprabha]

gocraft/work is a pretty outdated library.

Agreed, I also tried xworker with asynq and machinery, worked with both.

every framework has different trade-offs, UI, and APIs.

True, machinery had flows like async orchestration which I didn't support in xworker.

Adopt a new library like https://github.com/riverqueue/river.

Sure, haven't tried it yet, but it looks promising! The choice of underlying storage is very subjective, for instance, my workflows primarily rely on redis and kafka only, for me, postgres is an added dependency.

We can build our own async worker framework.

Decoupling the frontend SDK from underlying implementation is a great idea in most cases, example: Otel, but I agree that it also comes with huge complexity, however, I don't want to deny the benefits, looks like river solves that, then I would wonder if we even need a wrapper altogether. Can we get away with river itself.

[sonnes]

Let us try adding redis backend to river