feat(new sink): Initial aws_kinesis_firehose sink
#1388
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Luke Steensen <[email protected]>
Signed-off-by: Luke Steensen <[email protected]>
Signed-off-by: Luke Steensen <[email protected]>
Signed-off-by: Luke Steensen <[email protected]>
Signed-off-by: Luke Steensen <[email protected]>
lukesteensen
added
type: new feature
domain: sinks
binarylogic
changed the title
aws_kinesis_firehose sink
Signed-off-by: binarylogic <[email protected]>
Signed-off-by: binarylogic <[email protected]>
Signed-off-by: binarylogic <[email protected]>
Signed-off-by: binarylogic <[email protected]>
LucioFranco
reviewed
Dec 19, 2019
src/sinks/aws_kinesis_firehose.rs
Outdated
|
|
||
| #[derive(Clone)] | ||
| pub struct KinesisFirehoseService { | ||
| client: Arc<KinesisFirehoseClient>, |
There was a problem hiding this comment.
Don't think we need to arc this https://docs.rs/rusoto_firehose/0.42.0/rusoto_firehose/struct.KinesisFirehoseClient.html#impl-Clone
LucioFranco
reviewed
Dec 19, 2019
| }; | ||
| let common = ElasticSearchCommon::parse_config(&config).expect("Config error"); | ||
|
|
||
| let client = reqwest::Client::builder() |
There was a problem hiding this comment.
I kinda wanna get rid of reqwest and just use the elastic libraries serde feature. But we can do that in the future.
🔥Signed-off-by: Luke Steensen <[email protected]>
|
This looks great! I wrote a suggestion about further improving it in #1407. |
binarylogic
added
type: feature
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #559
Closes #1062
This is another stab at an
aws_kinesis_firehosesink. It's mostly a copied and modified version of theaws_kinesis_streamssinks, with some additional copying from theelasticsearchsink test setup (since we configure it to write to ES for the test).There's probably a good bit of deduplication we could do between the two sinks (and ES tests), but this PR defers all of that in favor of getting something working.
I initially tried to test against an S3 delivery stream, which we should do at some point, but was foiled by some combination of localstack's S3 implementation and rusoto (localstack/localstack#1647).
One thing that's unclear is whether or not we should add an option to encode newlines for delivery streams aimed at S3. I have not looked at how the objects look without it.
External refs: fluent/fluent-bit#2485