[aws-for-fluent-bit] Add helm chart high performance kinesis stream plugin #951

changhyuni · 2023-05-25T06:35:19Z

Issue

Description of changes

reference document: https://github.com/aws/amazon-kinesis-streams-for-fluent-bit#how-can-i-migrate-to-the-higher-performance-plugin

Added high performance kinesis stream plugin support for aws-for-fluent-bit Helm Chart
Updated ConfigMap and values.yaml
Updated README doc for Helm Chart

Checklist

Added/modified documentation as required (such as the README.md for modified charts)
Incremented the chart version in Chart.yaml for the modified chart(s)
Manually tested. Describe what testing was done in the testing section below
Make sure the title of the PR is a good description that can go into the release notes

Testing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

PettitWesley · 2023-05-25T19:46:52Z

stable/aws-for-fluent-bit/README.md

@@ -91,6 +91,15 @@ helm delete aws-for-fluent-bit --namespace kube-system
 | `firehose.timeKey` | Add the timestamp to the record under this key. By default the timestamp from Fluent Bit will not be added to records sent to Kinesis. | |
 | `firehose.timeKeyFormat` | strftime compliant format string for the timestamp; for example, `%Y-%m-%dT%H:%M:%S%z`. This option is used with `time_key`. | |
 | `firehose.extraOutputs` | Append extra outputs with value | `""` |
+| `kinesis_streams.enabled` | Whether this plugin should be enabled or not, [details](https://github.com/aws/amazon-kinesis-streams-for-fluent-bit) | `false` | ✔


link the C docs, not go docs: https://docs.fluentbit.io/manual/pipeline/outputs/kinesis

PettitWesley

Thank you for adding this!! Few comments/fixes.

PettitWesley · 2023-05-25T19:48:36Z

stable/aws-for-fluent-bit/README.md

+| `kinesis_streams.enabled` | Whether this plugin should be enabled or not, [details](https://github.com/aws/amazon-kinesis-streams-for-fluent-bit) | `false` | ✔
+| `kinesis_streams.region` | The AWS region. | ✔
+| `kinesis_streams.stream` | The name of the Kinesis Streams Delivery Stream that you want log records send to. | ✔
+| `kinesis_streams.sts_endpoint` | Custom endpoint for the STS API. `kinesis_streams.sts_endpoint`. | |


This seems to be missing a few options:

endpoint

role_arn

@PettitWesley Thnak you for review!
I updated readme your comment.

PettitWesley · 2023-05-26T18:27:21Z

stable/aws-for-fluent-bit/README.md

+| `kinesis_streams.sts_endpoint` | Custom endpoint for the STS API. `kinesis_streams.sts_endpoint`. | |
+| `kinesis_streams.time_key` | Add the timestamp to the record under this key. By default the timestamp from Fluent Bit will not be added to records sent to Kinesis. | |
+| `kinesis_streams.time_key_format` |  strftime compliant format string for the timestamp; for example, the default is `%Y-%m-%dT%H:%M:%S`. Supports millisecond precision with `%3N` and supports nanosecond precision with `%9N` and `%L`; for example, adding `%3N` to support millisecond `%Y-%m-%dT%H:%M:%S.%3N`. This option is used with `time_key`. | |
+| `kinesis_streams.log_key` | By default, the whole log record will be sent to Kinesis. If you specify a key name with this option, then only the value of that key will be sent to Kinesis. For example, if you are using the Fluentd Docker log driver, you can specify `log_key` `log` and only the log message will be sent to Kinesis. | |


nit: Please use log_key log same formatting as the public docs do to make it clear

@PettitWesley i fixed readme thank you.

PettitWesley · 2023-05-31T21:12:03Z

stable/aws-for-fluent-bit/README.md

@@ -91,6 +91,17 @@ helm delete aws-for-fluent-bit --namespace kube-system
 | `firehose.timeKey` | Add the timestamp to the record under this key. By default the timestamp from Fluent Bit will not be added to records sent to Kinesis. | |
 | `firehose.timeKeyFormat` | strftime compliant format string for the timestamp; for example, `%Y-%m-%dT%H:%M:%S%z`. This option is used with `time_key`. | |
 | `firehose.extraOutputs` | Append extra outputs with value | `""` |
+| `kinesis_streams.enabled` | It has all the core features of the `aws/amazon-kinesis-streams-for-fluent-bit` Golang Fluent Bit plugin released in 2019. The Golang plugin was named `kinesis`; this new high performance and highly efficient kinesis plugin is called `kinesis_streams` to prevent conflicts/confusion, [details](https://docs.fluentbit.io/manual/pipeline/outputs/kinesis) | `false` | ✔


aws/amazon-kinesis-streams-for-fluent-bit => this could be a link

PettitWesley

Please make the one readme change to add a link, then please confirm for me that you have tested this, and then we should be good to go

PettitWesley · 2023-05-31T21:13:00Z

I may forget about this PR, @ mention me to remind me.

changhyuni · 2023-06-01T01:49:38Z

@PettitWesley fixed!
What do you think about the url name "detail"?

PettitWesley

@changhyuni thanks, I think the link "details" title is fine... you could also change it to "documentation" instead

PettitWesley · 2023-06-01T02:31:14Z

@changhyuni once we get to the final version, can you please paste the results of generating the config map from an example with values.yaml containing all kinesis_streams config options as evidence of testing. Then I can approve.

PettitWesley · 2023-06-01T17:27:57Z

Please @ me when ready, I have a lot on my backlog and may forget this PR otherwise

changhyuni · 2023-07-07T12:16:34Z

@PettitWesley I'm sorry for replying late
I tested it with my company account and it works well
I erased the company information

PettitWesley · 2023-07-10T17:39:15Z

stable/aws-for-fluent-bit/README.md

+| `kinesis_streams.stream` | The name of the Kinesis Streams Delivery Stream that you want log records send to. | ✔
+| `kinesis_streams.endpoint` | Specify a custom endpoint for the Kinesis Streams API. | |
+| `kinesis_streams.role_arn` | ARN of an IAM role to assume (for cross account access). | |
+| `kinesis_streams.sts_endpoint` | Custom endpoint for the STS API. `kinesis_streams.sts_endpoint`. | |


Custom endpoint for the STS API. kinesis_streams.sts_endpoint

Why add the config option name again? kinesis_streams.sts_endpoint

Its redundant and not done for any other option.

PettitWesley · 2023-07-10T17:40:07Z

@changhyuni CI failed, looks like you need to bump chart version again, we did a release recently.

changhyuni · 2023-07-12T11:30:04Z

@PettitWesley thanks you!
i fixed it!

PettitWesley · 2023-07-12T22:59:32Z

stable/aws-for-fluent-bit/README.md

@@ -91,6 +91,17 @@ helm delete aws-for-fluent-bit --namespace kube-system
 | `firehose.timeKey` | Add the timestamp to the record under this key. By default the timestamp from Fluent Bit will not be added to records sent to Kinesis. | |
 | `firehose.timeKeyFormat` | strftime compliant format string for the timestamp; for example, `%Y-%m-%dT%H:%M:%S%z`. This option is used with `time_key`. | |
 | `firehose.extraOutputs` | Append extra outputs with value | `""` |
+| `kinesis_streams.enabled` | It has all the core features of the [documentation](https://github.com/aws/amazon-kinesis-streams-for-fluent-bit) Golang Fluent Bit plugin released in 2019. The Golang plugin was named `kinesis`; this new high performance and highly efficient kinesis plugin is called `kinesis_streams` to prevent conflicts/confusion, [details](https://docs.fluentbit.io/manual/pipeline/outputs/kinesis) | `false` | ✔


This reads a bit awkward. I'd do:

"Enable the high performance kinesis plugin; it has all the core features of the Golang Fluent Bit plugin..

PettitWesley · 2023-07-12T23:02:02Z

@changhyuni sorry but it says there are still conflicts...

I think you need a rebase:

git checkout master
git pull
git checkout update/aws-for-fluent-bit # feature branch
git rebase master
git push --force

Co-authored-by: eks-bot <[email protected]>

- 2.28.4 --> stable (less manual updates) - version 0.1.25 --> 0.1.26 - update readme

Co-authored-by: eks-bot <[email protected]>

… complaints aws/aws-for-fluent-bit#694 Signed-off-by: Wesley Pettit <[email protected]>

changhyuni · 2023-07-13T05:14:17Z

@PettitWesley thanks you,
I solved it

PettitWesley · 2023-07-13T17:27:12Z

@changhyuni Sorry it still says I can't merge it. Sometimes Github UI rebase does not work and you need to rebase on command line: #951 (comment)

Add template high performance kinesis stream plugin

fdc7876

changhyuni requested review from PettitWesley and hossain-rayhan as code owners May 25, 2023 06:35

PettitWesley reviewed May 25, 2023

View reviewed changes

PettitWesley suggested changes May 25, 2023

View reviewed changes

june.i added 2 commits May 26, 2023 14:37

update readme.md

1ff8e61

update readme kinesis streams url

4af7cda

PettitWesley reviewed May 26, 2023

View reviewed changes

PettitWesley reviewed May 31, 2023

View reviewed changes

update reademe

84ebee0

changhyuni force-pushed the update/aws-for-fluent-bit branch from 99df35d to 84ebee0 Compare June 1, 2023 01:43

june.i added 2 commits June 1, 2023 10:44

fix typo

b0e43ad

rename readme.md url name

f190f49

PettitWesley reviewed Jun 1, 2023

View reviewed changes

fix typo

0702f25

PettitWesley reviewed Jul 10, 2023

View reviewed changes

PettitWesley previously approved these changes Jul 10, 2023

View reviewed changes

changhyuni dismissed PettitWesley’s stale review via d067dd5 July 12, 2023 11:29

PettitWesley reviewed Jul 12, 2023

View reviewed changes

PettitWesley previously approved these changes Jul 12, 2023

View reviewed changes

chart updates for VPC CNI v1.13.0 release (aws#952)

e9d951d

ysdongAmazon and others added 9 commits July 13, 2023 14:09

appmesh-controller: v1.12.1 (aws#954)

8a709f3

Co-authored-by: eks-bot <[email protected]>

fix(fluentbit): rewrite erroneous ident

3924142

fix(fluentbit): rewrite erroneous ident

9180a69

Bumping aws-for-fluent-bit helm chart version

377cb10

- 2.28.4 --> stable (less manual updates) - version 0.1.25 --> 0.1.26 - update readme

VPC CNI updates for v1.13.2 release (aws#962)

27118eb

aws-load-balancer-controller: v2.5.3 (aws#968)

ca97c94

Co-authored-by: eks-bot <[email protected]>

aws-for-fluent-bit: switch to hardcode stable version to address user…

ff9ab6e

… complaints aws/aws-for-fluent-bit#694 Signed-off-by: Wesley Pettit <[email protected]>

fix document

49fd309

fix chart version

365cc1b

changhyuni force-pushed the update/aws-for-fluent-bit branch from d067dd5 to 365cc1b Compare July 13, 2023 05:09

changhyuni requested review from jayanthvn, kishorj, M00nF1sh, a team and bwagner5 as code owners July 13, 2023 05:09

Merge branch 'master' into update/aws-for-fluent-bit

a75beb5

changhyuni dismissed PettitWesley’s stale review via a75beb5 July 13, 2023 05:13

PettitWesley approved these changes Jul 13, 2023

View reviewed changes

PettitWesley merged commit 41f2640 into aws:master Jul 26, 2023
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[aws-for-fluent-bit] Add helm chart high performance kinesis stream plugin #951

[aws-for-fluent-bit] Add helm chart high performance kinesis stream plugin #951

changhyuni commented May 25, 2023 •

edited

Loading

PettitWesley May 25, 2023

PettitWesley left a comment

PettitWesley May 25, 2023

changhyuni May 26, 2023 •

edited

Loading

PettitWesley May 26, 2023

changhyuni May 28, 2023

PettitWesley May 31, 2023

PettitWesley left a comment

PettitWesley commented May 31, 2023

changhyuni commented Jun 1, 2023

PettitWesley left a comment

PettitWesley commented Jun 1, 2023

PettitWesley commented Jun 1, 2023

changhyuni commented Jul 7, 2023

PettitWesley Jul 10, 2023

PettitWesley Jul 10, 2023

PettitWesley commented Jul 10, 2023

changhyuni commented Jul 12, 2023

PettitWesley Jul 12, 2023

PettitWesley commented Jul 12, 2023 •

edited

Loading

changhyuni commented Jul 13, 2023

PettitWesley commented Jul 13, 2023

[aws-for-fluent-bit] Add helm chart high performance kinesis stream plugin #951

[aws-for-fluent-bit] Add helm chart high performance kinesis stream plugin #951

Conversation

changhyuni commented May 25, 2023 • edited Loading

Issue

Description of changes

Checklist

Testing

Choose a reason for hiding this comment

PettitWesley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

changhyuni May 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PettitWesley left a comment

Choose a reason for hiding this comment

PettitWesley commented May 31, 2023

changhyuni commented Jun 1, 2023

PettitWesley left a comment

Choose a reason for hiding this comment

PettitWesley commented Jun 1, 2023

PettitWesley commented Jun 1, 2023

changhyuni commented Jul 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PettitWesley commented Jul 10, 2023

changhyuni commented Jul 12, 2023

Choose a reason for hiding this comment

PettitWesley commented Jul 12, 2023 • edited Loading

changhyuni commented Jul 13, 2023

PettitWesley commented Jul 13, 2023

changhyuni commented May 25, 2023 •

edited

Loading

changhyuni May 26, 2023 •

edited

Loading

PettitWesley commented Jul 12, 2023 •

edited

Loading