Skip to content

Commit

Permalink
Update Getting Started (#494)
Browse files Browse the repository at this point in the history
  • Loading branch information
@m110
m110 authored Sep 25, 2024
1 parent 400f4c5 commit 75d4f9c
Show file tree
Hide file tree
Showing 4 changed files with 97 additions and 88 deletions.
175 changes: 88 additions & 87 deletions docs/content/docs/getting-started.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,51 +2,63 @@
title = "Getting started"
description = "Watermill up and running"
weight = -9999
draft = false
draft = false
toc = true
bref = "Watermill up and running"
type = "docs"
+++

### What is Watermill?

Watermill is a Golang library for working efficiently with message streams. It is intended for building event-driven
applications. It can be used for event sourcing, RPC over messages, sagas, and whatever else comes to your mind.
You can use conventional pub/sub implementations like Kafka or RabbitMQ, but also HTTP or MySQL binlog, if that fits your use case.

It comes with a set of Pub/Sub implementations and can be easily extended by your own.
Watermill is a Go library for working with message streams.
You can use it to build event-driven systems with popular Pub/Sub implementations like Kafka or RabbitMQ, as well as HTTP or Postgres if that fits your use case.
It comes with a set of Pub/Sub implementations and can be easily extended.

Watermill also ships with standard middlewares like instrumentation, poison queue, throttling, correlation,
and other tools used by every message-driven application.

### Why use Watermill?

With more projects adopting the microservices pattern over recent years, we realized that synchronous communication
is not always the right choice. Asynchronous methods started to grow as a new standard way to communicate.
When using microservices, synchronous communication is not always the right choice.
Asynchronous methods became a new standard way to communicate.

While there are many tools and libraries for synchronous communication, like HTTP, correctly setting up
a message-oriented project can be challenging. There are many different message queues and streaming systems,
each with different features, client libraries, and APIs.

But while there's a lot of existing tooling for synchronous integration patterns (e.g. HTTP), correctly setting up
a message-oriented project can be a challenge. There's a lot of different message queues and streaming systems,
each with different features and client library API.
Watermill aims to be the standard messaging library for Go, hiding all that complexity behind an API that is easy to understand.
It provides all you need to build an application based on events or other asynchronous patterns.

Watermill aims to be the standard messaging library for Go, hiding all that complexity behind an API that is easy to
understand. It provides all you might need for building an application based on events or other asynchronous patterns.
After looking at the examples, you should be able to quickly integrate Watermill with your project.
**Watermill is NOT a framework**.
It's a lightweight library that's easy to plug in or remove from your project.

### Install

```bash
go get -u github.com/ThreeDotsLabs/watermill
```

### One Minute Background
### One-Minute Background

The basic idea behind event-driven applications stays always the same: listen for incoming messages and react to them.
The idea behind event-driven applications is always the same: listen to and react to incoming messages.
Watermill supports this behavior for multiple [publishers and subscribers]({{< ref "/pubsubs" >}}).

The core part of Watermill is the [*Message*]({{< ref "/docs/message" >}}). It is as important as `http.Request`
is for the `http` package. Most Watermill features use this struct in some way.
The core part of Watermill is the [*Message*]({{< ref "/docs/message" >}}).
It is what `http.Request` is for the `net/http` package.
Most Watermill features work with this struct.

Watermill provides a few APIs for working with messages.
They build on top of each other, each step providing a higher-level API:

* At the bottom, the `Publisher` and `Subscriber` interfaces. It's the "raw" way of working with messages. You get full control, but also need to handle everything yourself.
* The `Router` is similar to HTTP routers you probably know. It introduces message handlers.
* The `CQRS` component adds generic handlers without needing to marshal and unmarshal messages yourself.

<img src="/img/pyramid.png" alt="Watermill components pyramid" style="width:100%;" />

Even though PubSub libraries come with complex features, for Watermill it's enough to implement two interfaces to start
## Publisher & Subscriber

Most Pub/Sub libraries come with complex features. For Watermill, it's enough to implement two interfaces to start
working with them: the `Publisher` and `Subscriber`.

```go
Expand All @@ -63,8 +75,9 @@ type Subscriber interface {

### Subscribing for Messages

Let's start with subscribing. `Subscribe` expects a topic name and returns a channel of incoming messages.
What _topic_ exactly means depends on the PubSub implementation.
`Subscribe` expects a topic name and returns a channel of incoming messages.
What _topic_ exactly means depends on the Pub/Sub implementation.
Usually, it needs to match the topic name used by the publisher.

```go
messages, err := subscriber.Subscribe(ctx, "example.topic")
Expand Down Expand Up @@ -95,15 +108,15 @@ See detailed examples below for supported PubSubs.
Running in Docker
{{% /collapse-toggle %}}
{{% collapse-box id="docker" %}}
The easiest way to run Watermill locally with Kafka is using Docker.
The easiest way to run Watermill locally with Kafka is by using Docker.

{{% load-snippet file="src-link/_examples/pubsubs/kafka/docker-compose.yml" type="yaml" %}}

The source should go to `main.go`.

To run, execute `docker-compose up` command.
To run, execute the `docker-compose up` command.

A more detailed explanation of how it is working (and how to add live code reload) can be found in [*Go Docker dev environment* article](https://threedots.tech/post/go-docker-dev-environment-with-go-modules-and-live-code-reloading/).
A more detailed explanation of how it works (and how to add live code reload) can be found in the [*Go Docker dev environment* article](https://threedots.tech/post/go-docker-dev-environment-with-go-modules-and-live-code-reloading/).

{{% /collapse-box %}}
{{< /collapse >}}
Expand All @@ -126,7 +139,7 @@ The easiest way to run Watermill locally with NATS is using Docker.

The source should go to `main.go`.

To run execute `docker-compose up` command.
To run, execute the `docker-compose up` command.

A more detailed explanation of how it is working (and how to add live code reload) can be found in [*Go Docker dev environment* article](https://threedots.tech/post/go-docker-dev-environment-with-go-modules-and-live-code-reloading/).
{{% /collapse-box %}}
Expand All @@ -145,7 +158,7 @@ A more detailed explanation of how it is working (and how to add live code reloa
Running in Docker
{{% /collapse-toggle %}}
{{% collapse-box id="gcloud-streaming-docker" %}}
You can run Google Cloud Pub/Sub emulator locally for development.
You can run the Google Cloud Pub/Sub emulator locally for development.

{{% load-snippet file="src-link/_examples/pubsubs/googlecloud/docker-compose.yml" type="yaml" %}}

Expand Down Expand Up @@ -209,10 +222,10 @@ A more detailed explanation of how it is working (and how to add live code reloa

### Creating Messages

Watermill doesn't enforce any message format. `NewMessage` expects a slice of bytes as the payload. You can use
strings, JSON, protobuf, Avro, gob, or anything else that serializes to `[]byte`.
Watermill doesn't enforce any message format. `NewMessage` expects a slice of bytes as the payload.
You can use strings, JSON, protobuf, Avro, gob, or anything else that serializes to `[]byte`.

The message UUID is optional, but recommended, as it helps with debugging.
The message UUID is optional but recommended for debugging.

```go
msg := message.NewMessage(watermill.NewUUID(), []byte("Hello, world!"))
Expand All @@ -232,121 +245,109 @@ if err != nil {
{{< tabs id="publishing" tabs="go-channel,kafka,nats-streaming,gcloud,amqp,sql" labels="Go Channel,Kafka,NATS Streaming,Google Cloud Pub/Sub,RabbitMQ (AMQP),SQL" >}}

{{% tabs-tab id="go-channel"%}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/go-channel/main.go" first_line_contains="go process(messages)" last_line_contains="publisher.Publish" padding_after="4" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/go-channel/main.go" first_line_contains="message.NewMessage" last_line_contains="publisher.Publish" padding_after="2" %}}
{{% /tabs-tab %}}

{{% tabs-tab id="kafka" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/kafka/main.go" first_line_contains="go process(messages)" last_line_contains="publisher.Publish" padding_after="4" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/kafka/main.go" first_line_contains="message.NewMessage" last_line_contains="publisher.Publish" padding_after="2" %}}
{{% /tabs-tab %}}

{{% tabs-tab id="nats-streaming" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/nats-streaming/main.go" first_line_contains="go process(messages)" last_line_contains="publisher.Publish" padding_after="4" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/nats-streaming/main.go" first_line_contains="message.NewMessage" last_line_contains="publisher.Publish" padding_after="2" %}}
{{% /tabs-tab %}}

{{% tabs-tab id="gcloud" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/googlecloud/main.go" first_line_contains="go process(messages)" last_line_contains="publisher.Publish" padding_after="4" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/googlecloud/main.go" first_line_contains="message.NewMessage" last_line_contains="publisher.Publish" padding_after="2" %}}
{{% /tabs-tab %}}

{{% tabs-tab id="amqp" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/amqp/main.go" first_line_contains="go process(messages)" last_line_contains="publisher.Publish" padding_after="4" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/amqp/main.go" first_line_contains="message.NewMessage" last_line_contains="publisher.Publish" padding_after="2" %}}
{{% /tabs-tab %}}

{{% tabs-tab id="sql" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/sql/main.go" first_line_contains="go process(messages)" last_line_contains="publisher.Publish" padding_after="4" %}}
{{% load-snippet-partial file="src-link/_examples/pubsubs/sql/main.go" first_line_contains="message.NewMessage" last_line_contains="publisher.Publish" padding_after="2" %}}
{{% /tabs-tab %}}

{{< /tabs >}}

### Using *Message Router*
### Router

[*Publishers and subscribers*]({{< ref "/docs/pub-sub" >}}) are rather low-level parts of Watermill.
In most cases, you'd usually want to use a high-level interface and features like [correlation, metrics, poison queue, retrying, throttling, etc.]({{< ref "/docs/messages-router#middleware" >}}).
[*Publishers and subscribers*]({{< ref "/docs/pub-sub" >}}) are the low-level parts of Watermill.
For most cases, you want to use a high-level API: [*Router*]({{< ref "/docs/messages-router" >}}) component.

You might want to send an Ack only if the message was processed successfully.
In other cases, you'll Ack immediately and then worry about processing.
Sometimes, you want to perform some action based on the incoming message, and publish another message in response.
#### Router configuration

To handle these requirements, there is a component named [*Router*]({{< ref "/docs/messages-router" >}}).
Start with configuring the router and adding plugins and middlewares.

### Example application of *Message Router*
The flow of the example application looks like this:
A middleware is a function executed for each incoming message.
You can use one of the existing ones for things like [correlation, metrics, poison queue, retrying, throttling, etc.]({{< ref "/docs/messages-router#middleware" >}}).
You can also create your own.

{{% render-md %}}
{{% load-snippet-partial file="src-link/_examples/basic/3-router/main.go" first_line_contains="message.NewRouter" last_line_contains="middleware.Recoverer," padding_after="1" %}}
{{% /render-md %}}

1. A message is produced on topic `incoming_messages_topic` every second.
2. `struct_handler` handler listens on `incoming_messages_topic`. When a message is received, the UUID is printed and a new message is produced on `outgoing_messages_topic`.
3. `print_incoming_messages` handler listens on `incoming_messages_topic` and prints the messages' UUID, payload and metadata.
4. `print_outgoing_messages` handler listens on `outgoing_messages_topic` and prints the messages' UUID, payload and metadata. Correlation ID should be the same as in the message on `incoming_messages_topic`.
#### Handlers

#### Router configuration
Set up handlers that the router uses.
Each handler independently handles incoming messages.

Start with configuring the router, adding plugins and middlewares.
Then set up handlers that the router will use. Each handler will independently handle messages.
A handler listens to messages from the given subscriber and topic.
Any messages returned from the handler function will be published to the given publisher and topic.

{{% render-md %}}
{{% load-snippet-partial file="src-link/_examples/basic/3-router/main.go" first_line_contains="package" last_line_contains="router.Run(ctx)" padding_after="4" %}}
{{% load-snippet-partial file="src-link/_examples/basic/3-router/main.go" first_line_contains="AddHandler returns" last_line_contains=")" padding_after="0" %}}
{{% /render-md %}}

#### Incoming messages
*Note: the example above uses one `pubSub` argument for both the subscriber and publisher.
It's because we use the `GoChannel` implementation, which is a simple in-memory Pub/Sub.*

The `struct_handler` consumes messages from `incoming_messages_topic`, so we are simulating incoming traffic by calling `publishMessages()` in the background.
Notice that we've added the `SetCorrelationID` middleware. A Correlation ID will be added to all messages produced by the router (it will be stored in metadata).
Alternatively, if you don't plan to publish messages from within the handler, you can use the simpler `AddNoPublisherHandler` method.

{{% render-md %}}
{{% load-snippet-partial file="src-link/_examples/basic/3-router/main.go" first_line_contains="func publishMessages" last_line_contains="time.Sleep(time.Second)" padding_after="2" %}}
{{% load-snippet-partial file="src-link/_examples/basic/3-router/main.go" first_line_contains="AddNoPublisherHandler" last_line_contains=")" padding_after="0" %}}
{{% /render-md %}}

#### Handlers

You may have noticed that there are two types of *handler functions*:
You can use two types of *handler functions*:

1. function `func(msg *message.Message) ([]*message.Message, error)`
2. method `func (c structHandler) Handler(msg *message.Message) ([]*message.Message, error)`
1. a function `func(msg *message.Message) ([]*message.Message, error)`
2. a struct method `func (c structHandler) Handler(msg *message.Message) ([]*message.Message, error)`

If your handler is a function without any dependencies, it's fine to use the first one.
The second option is useful when your handler requires some dependencies like database handle, a logger, etc.
Use the first one if your handler is a function without any dependencies.
The second option is useful when your handler requires dependencies such as a database handle or a logger.

{{% render-md %}}
{{% load-snippet-partial file="src-link/_examples/basic/3-router/main.go" first_line_contains="func printMessages" last_line_contains="return message.Messages{msg}, nil" padding_after="3" %}}
{{% /render-md %}}

#### Done!

You can run this example by `go run main.go`.

You've just created your first application with Watermill. You can find the full source in [/_examples/basic/3-router/main.go](https://github.com/ThreeDotsLabs/watermill/blob/master/_examples/basic/3-router/main.go).
The complete example's source can be found at [/_examples/basic/3-router/main.go](https://github.com/ThreeDotsLabs/watermill/blob/master/_examples/basic/3-router/main.go).

### Logging

To see Watermill's logs, you have to pass any logger that implements the [LoggerAdapter](https://github.com/ThreeDotsLabs/watermill/blob/master/log.go).
To see Watermill's logs, pass any logger that implements the [LoggerAdapter](https://github.com/ThreeDotsLabs/watermill/blob/master/log.go).
For experimental development, you can use `NewStdLogger`.

### Testing

Watermill provides [a set of test scenarios](https://github.com/ThreeDotsLabs/watermill/blob/master/pubsub/tests/test_pubsub.go)
that any Pub/Sub implementation can use. Each test suite needs to declare what features it supports and how to construct a new Pub/Sub.
These scenarios check both basic usage and more uncommon use cases. Stress tests are also included.

### Deployment

Watermill is not a framework. We don't enforce any type of deployment and it's totally up to you.
## What's next?

### What's next?
For more details, see [documentation topics]({{< ref "/docs" >}}).

For more detailed documentation check [documentation topics]({{< ref "/docs" >}}).
See the [CQRS component](/docs/cqrs) for another high-level API.

#### Examples
## Examples

Check out the [examples](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples) that will show you how to start using Watermill.

The recommended entry point is [Your first Watermill application](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/basic/1-your-first-app).
It contains the entire environment in `docker-compose.yml`, including Golang and Kafka, which you can run with one command.
It contains the entire environment in `docker-compose.yml`, including Go and Kafka, which you can run with one command.

After that, you can see the [Realtime feed](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/basic/2-realtime-feed) example.
It uses more middlewares and contains two handlers. There is also a separate application for publishing messages.
It uses more middlewares and contains two handlers.

For a different subscriber implementation, namely **HTTP**, refer to the [receiving-webhooks](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/real-world-examples/receiving-webhooks) example. It is a very simple application that saves webhooks to Kafka.
For a different subscriber implementation (**HTTP**), see the [receiving-webhooks](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/real-world-examples/receiving-webhooks) example.
It is a straightforward application that saves webhooks to Kafka.

Full list of examples can be found in the project's [README](https://github.com/ThreeDotsLabs/watermill#examples).
You can find the complete list of examples in the [README](https://github.com/ThreeDotsLabs/watermill#examples).

#### Support
## Support

If anything is not clear, feel free to use any of our [support channels]({{< ref "/support" >}}), we will be glad to help.
If anything is not clear, feel free to use any of our [support channels]({{< ref "/support" >}}); we will be glad to help.
6 changes: 6 additions & 0 deletions docs/content/docs/pub-sub-implementing.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@ To add support for a custom Pub/Sub, you have to implement both `message.Publish
{{% load-snippet-partial file="src-link/message/pubsub.go" first_line_contains="type Publisher interface" last_line_contains="type SubscribeInitializer" padding_after="0" %}}
{{% /render-md %}}

### Testing

Watermill provides [a set of test scenarios](https://github.com/ThreeDotsLabs/watermill/blob/master/pubsub/tests/test_pubsub.go)
that any Pub/Sub implementation can use. Each test suite needs to declare what features it supports and how to construct a new Pub/Sub.
These scenarios check both basic usage and more uncommon use cases. Stress tests are also included.

### TODO list

Here are a few things you shouldn't forget about:
Expand Down
Loading

0 comments on commit 75d4f9c

Please sign in to comment.