Skip to content
This repository has been archived by the owner on Mar 13, 2024. It is now read-only.

feat(examples): add generate example #26

Merged
merged 1 commit into from
Jan 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ The following examples demonstrate how to ingest events from various sources int
The examples use the custom Benthos distribution in this repository.

- [Database](database/)
- [Generate](generate/) random events
- [HTTP server](http-server/) (forwarding events to OpenMeter)
- [Kubernetes Pod execution time](kubernetes-pod-exec-time/)

Expand Down
75 changes: 75 additions & 0 deletions examples/generate/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# Generate

This example is based on [our earlier blog post](https://openmeter.io/blog/testing-stream-processing) about testing OpenMeter with Benthos.

The input in this case generates random events (mimicking API calls).

## Table of Contents <!-- omit from toc -->

- [Prerequisites](#prerequisites)
- [Launch the example](#launch-the-example)
- [Checking events](#checking-events)
- [Advanced configuration](#advanced-configuration)
- [Production use](#production-use)

## Prerequisites

Go to the [GitHub Releases](https://github.com/openmeterio/benthos-openmeter/releases/latest) page and download the latest `benthos` binary for your platform.

Check out this repository if you want to run the example locally:

```shell
git clone https://github.com/openmeterio/benthos-openmeter.git
cd benthos-openmeter/examples/generate
```

Create a new `.env` file and add the details of your OpenMeter instance:

```shell
cp .env.dist .env
# edit .env and fill in the details
```

> [!TIP]
> Tweak other options in the `.env` file to change the behavior of the example.

Create a meter in OpenMeter with the following details:

- Event type: `api-calls`
- Aggregation: `SUM`
- Value property: `$.duration_ms`
- Group by (optional):
- `method`: `$.method`
- `path`: `$.path`
- `region`: `$.region`
- `zone`: `$.zone`

> [!TIP]
> Read more about creating a meter in the general examples [README](../README.md#Create-a-meter).

## Launch the example

Launch the example:

```shell
export OPENMETER_TOKEN=<YOUR TOKEN>
benthos -c config.yaml
```

> [!WARNING]
> By default the example generates 1000 events per second.

## Checking events

Read more in the general examples [README](../README.md#Checking-events-in-OpenMeter).

## Advanced configuration

Check out the configuration files and the [Benthos documentation](https://www.benthos.dev/docs/about) for more details.

## Production use

We are actively working on improving the documentation and the examples.
In the meantime, feel free to contact us [in email](https://us10.list-manage.com/contact-form?u=c7d6a96403a0e5e19032ee885&form_id=fe04a7fc4851f8547cfee56763850e95) or [on Discord](https://discord.gg/nYH3ZQ3Xzq).

We are more than happy to help you set up OpenMeter in your production environment.
57 changes: 57 additions & 0 deletions examples/generate/config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
input:
generate:
count: ${SEEDER_COUNT:0}
interval: "${SEEDER_INTERVAL:1s}"
batch_size: ${SEEDER_BATCH_SIZE:1}
mapping: |
let max_subjects = ${SEEDER_MAX_SUBJECTS:100}

let event_type = "api-calls"
let source = "api-gateway"
let methods = ["GET", "POST"]
let paths = ["/", "/about", "/contact", "/pricing", "/docs"]
let regions = ["us-east-1", "us-west-1", "us-east-2", "us-west-2"]
let zoneSuffixes = ["a", "b", "c", "d"]

let subject = "customer-%d".format(random_int(seed: timestamp_unix_nano()) % $max_subjects)
let time = (now().ts_sub_iso8601("P3D").ts_unix() + random_int(min: 60, max: 60 * 60 * 24 * 3)).ts_format()

let method = $methods.index(random_int(seed: timestamp_unix_nano()) % $methods.length())
let path = $paths.index(random_int(seed: timestamp_unix_nano()) % $paths.length())
let region = $regions.index(random_int(seed: timestamp_unix_nano()) % $regions.length())
let zone = "%s%s".format($region, $zoneSuffixes.index(random_int(seed: timestamp_unix_nano()) % $zoneSuffixes.length()))
let duration = random_int(seed: timestamp_unix_nano(), max: 1000)

root = {
"id": uuid_v4(),
"specversion": "1.0",
"type": $event_type,
"source": $source,
"subject": $subject,
"time": $time,
"data": {
"method": $method,
"path": $path,
"region": $region,
"zone": $zone,
"duration_ms": $duration,
},
}

output:
switch:
cases:
- check: ""
continue: true
output:
openmeter:
url: "${OPENMETER_URL:https://openmeter.cloud}"
token: "${OPENMETER_TOKEN:}"
batching:
count: ${BATCH_SIZE:20}
period: ${BATCH_PERIOD:}

- check: '"${SEEDER_LOG:false}" == "true"'
output:
stdout:
codec: lines