Merge pull request #10 from odpf/release-prep

Release prep

.github/workflows/package.yaml

@@ -1,12 +1,12 @@
-name: Package
+name: Release
 on:
   push:
     # Sequence of patterns matched against refs/tags
     tags:
       - "v*.*.*" # Push events to matching v*, i.e. v1.0, v20.15.10
 
 jobs:
-  docker:
+  push-to-docker-hub:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout code
@@ -35,3 +35,13 @@ jobs:
             docker.pkg.github.com/${{ github.repository }}/raccoon:${{ steps.get_version.outputs.version-without-v }}
             odpf/raccoon:latest
             odpf/raccoon:${{ steps.get_version.outputs.version-without-v }}
+  create-gh-release:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Release
+        uses: softprops/action-gh-release@v1
+        with:
+          body_path: CHANGELOG.md
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

docs/concepts/structure.md

@@ -1,2 +1,22 @@
 # Structure
+This document describes high-level code structure of the project. You'll find this part useful when you want to get started to contribute to Raccoon.
 
+## Highlevel View
+The core structure of Raccoon is the server itself. After the server is started, data flows from `websocket` to `worker` to `publisher`. `websocket` manages websocket server, handle incoming connection, and incoming request. `worker` acts as a buffer and interface for various types of server and publisher down the roadmap. `publisher` contains logic to publish the events to the downstream pipeline.
+![high-level](../assets/structure.svg)
+All the components above are initialized on `app`. `app` package is the starting point of Raccoon.
+## Code Map
+This section talks briefly about the content of various important packages. You can use this to guess the code location when you need to make changes.
+### `websocket`
+Contains server-related code along with request/response handlers and [connection management](architecture.md#connections).
+### `worker`
+Buffer from when the events are processed and before events are published. This will also act as interface that connects server and publisher when in the future. Currently, `worker` is tightly coupled with `websocket` server and `kafka` publisher.
+### `publisher`
+Does the actual publishing to the downstream pipeline. Currently, only support Kafka publisher.
+### `app`
+The starting point of Raccoon. It initializes server, worker, publisher, and other components that require initialization like a metric reporter.
+### `config`
+Load and store application configurations. Contains mapping of environment configuration with configuration on the code.
+## Code Generation
+### Request, Response, and Events Proto
+Raccoon depends on [Proton](https://github.com/odpf/proton/tree/main/odpf/raccoon) repository. Proton is a repository to store all ODPF Protobuf files. Code to serde the request and response are generated using Protobuf. You can check how the code is generated on `Makefile`.

docs/contribute/development.md

@@ -1,2 +1,29 @@
 # Development Guide
 
+This guide is targeted at developers looking to contribute to Raccoon.
+
+## Making a Pull Request
+
+#### Incorporating upstream changes from main
+
+Our preference is the use of `git rebase` instead of `git merge` : `git pull -r`
+
+#### Signing commits
+
+Commits have to be signed before they are allowed to be merged into the codebase:
+
+```bash
+# Include -s flag to signoff
+git commit -s -m "My first commit"
+```
+
+#### Good practices to keep in mind
+
+- Follow [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/#summary) while composing our commit messages.
+- Add `WIP:` to PR name if more work needs to be done prior to review
+- Avoid `force-pushing` as it makes reviewing difficult
+
+**Managing CI-test failures**
+
+- GitHub runner tests
+  - Click `checks` tab to analyse failed tests

docs/contribute/release.md

@@ -1,2 +1,23 @@
-# Release Process
+# Release
+For maintainers, please read the sections below as a guide to create a new release.
+## Create A New Release
+Please follow these steps to create a new release:
+- Update `version.txt` file
+- Generate changelog from commits by using [conventional-changelog-cli](https://www.npmjs.com/package/conventional-changelog-cli#quick-start)
+  ```sh
+  $ conventional-changelog -s -p conventionalcommits -i CHANGELOG.md
+  ```
+- Commit `version.txt` and `CHANGELOG.md` together and mark the commit with the release tag. Make sure the release tag and `version.txt` are the same.
+  ```sh
+  $ git add version.txt CHANGELOG.md
+  $ git commit -m "docs: update changelog and version for vM.m.p"
+  $ git tag vM.m.p
+  ```
+- Push the commit and the tag. Release action will trigger to publish docker image and create GitHub release.
 
+## Important Notes
+- Raccoon release tags follow [SEMVER](https://semver.org/) convention.
+- Github workflow is used to build and push the built docker image to Docker hub.
+- A release is triggered when a github tag of format `vM.m.p` is pushed. After the release job is succeeded, a docker image of
+format `M.m.p` is pushed to docker hub.
+- Release tags should only point to main branch

docs/guides/monitoring.md

@@ -3,6 +3,65 @@
 Raccoon provides monitoring for server connection, Kafka publisher, resource usage, and event delivery. Reference for available metrics is available [here](../reference/metrics.md). The metrics are reported using [Statsd](https://www.datadoghq.com/blog/statsd/) protocol.
 
 ## How To Setup
+```
+TL;DR
+- Run Statsd supported metric collector
+- Configure `METRIC_STATSD_ADDRESS` on Raccoon to send to the metric collector
+- Visualize and create alerting from the collected metrics
+```
+Generally, you can follow the steps above and use any metric collector that supports statsd like [Telegraf](https://www.influxdata.com/blog/getting-started-with-sending-statsd-metrics-to-telegraf-influxdb/) or [Datadog](https://docs.datadoghq.com/developers/dogstatsd/?tab=hostagent).
 
-First, you need to configure the metric collector that support Statsd. You can use [Telegraf](https://www.influxdata.com/blog/getting-started-with-sending-statsd-metrics-to-telegraf-influxdb/) or [Datadog](https://docs.datadoghq.com/developers/dogstatsd/?tab=hostagent) for example. You will have port which the metric collector listen. After you have the collector with port configured, you need to set [METRIC\_STATSD\_ADDRESS](https://github.com/odpf/raccoon/tree/21fd01b89db3fc3e3b947d545a1c47328fa28366/docs/reference/configuration.md) to match the metric collector port. What left is to visualize the metrics once you collect the data. For example, you can use [Chronograf](https://www.influxdata.com/time-series-platform/chronograf/) or [Grafana](https://github.com/grafana/grafana/) with Telegraf.
+In this section, we will cover a setup example using [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/), [Influx](https://www.influxdata.com/), [Kapacitor](https://www.influxdata.com/time-series-platform/kapacitor/), and [Grafana](https://grafana.com/) stack based on the steps above.
 
+**Run Statsd Supported Metric Collector**
+To enable statsd on Telegraf you need to enable statsd input on `telegraf.conf` file. Following are default configurations that you can add based on statsd input [readme](https://github.com/influxdata/telegraf/blob/master/plugins/inputs/statsd/README.md).
+```
+[[inputs.statsd]]
+  protocol = "udp"
+  max_tcp_connections = 250
+  tcp_keep_alive = false
+  service_address = ":8125"
+
+  delete_gauges = true
+  delete_counters = true
+  delete_sets = true
+  delete_timings = true
+
+  percentiles = [50.0, 90.0, 99.0, 99.9, 99.95, 100.0]
+
+  metric_separator = "_"
+
+  parse_data_dog_tags = false
+  datadog_extensions = false
+  datadog_distributions = false
+
+  allowed_pending_messages = 10000
+  percentile_limit = 1000
+
+[[outputs.influxdb]]
+  urls = ["http://127.0.0.1:8086"]
+  database = "raccoon"
+  retention_policy = "autogen"
+  write_consistency = "any"
+  timeout = "5s"
+```
+
+**Configure Raccoon To Send To The Metric Collector**
+After you have the collector with the port configured, you need to set [METRIC\_STATSD\_ADDRESS](https://odpf.gitbook.io/raccoon/reference/configurations#metric_statsd_address) to match the metric collector address. Suppose you deploy the telegraf using the default configuration above as sidecar or in localhost, you need to set the value to `:8125`.
+
+**Visualize And Create Alerting From The Collected Metrics**
+Now that you have Raccoon and Telegraf as metric collector set, next is to use the metrics reported. You may notice that the Telegraf config above contains `outputs.influxdb`. That config will send the metric received to influxdb. Make sure you have influx service accessible from the configured URL.
+You can visualize the metrics using Grafana. To do that, you need to [add influx datasource](https://www.influxdata.com/blog/how-grafana-dashboard-influxdb-flux-influxql/) to make the data available on Grafana. After that you can use the data to You can visualize the metrics using Grafana. To do that, you need to [add influx datasource](https://www.influxdata.com/blog/how-grafana-dashboard-influxdb-flux-influxql/) to make the data available on Grafana. After that, you can use the data to [make dashboard](https://grafana.com/docs/grafana/latest/datasources/influxdb/#influxql-query-editor)
+.
+
+## Metrics Usages
+
+Following are key monitoring statistics that you can infer from Raccoon metrics. Refer to this section to get a clue of how to build alerting, dashboard, or do analysis on the metrics.
+### Data Loss
+To infer data loss, you can count [`kafka_messages_delivered_total`](https://odpf.gitbook.io/raccoon/reference/metrics#kafka_messages_delivered_total) with tag `success=false`. You can also infer loss rate by calculating 
+> count(`kafka_messages_delivered_total` `success=false`) / count(`kafka_messages_delivered_total`)
+### Latency
+Raccoon provides fine-grained metrics that denote latency. That gives clues as to where to look in case something goes wrong. In summary, these are key metrics for latency:
+- [`event_processing_duration_milliseconds`](https://odpf.gitbook.io/raccoon/reference/metrics#event_processing_duration_milliseconds) This metrics denotes overall latency. You need to look at other latency metrics to find the root cause when this metric is high.
+- [`server_processing_latency_milliseconds`](https://odpf.gitbook.io/raccoon/reference/metrics#server_processing_latency_milliseconds) Correlate this metric with `event_processing_duration_milliseconds` to infer whether the issue is with Raccoon itself, or something wrong with the network, or the way [sent_time](https://github.com/odpf/proton/blob/main/odpf/raccoon/EventRequest.proto#L43) is generated.
+- [`worker_processing_duration_milliseconds`](https://odpf.gitbook.io/raccoon/reference/metrics#worker_processing_duration_milliseconds) High value of this metric indicates that the publisher is slow or can't keep up.