feat(elasticsearch) add new elasticsearch_v8 output #3160
Conversation
ooesili
added
enhancement
outputs
|
|
| @@ -35,6 +35,7 @@ import ( | |||
| _ "github.com/redpanda-data/connect/v4/public/components/dgraph" | |||
| _ "github.com/redpanda-data/connect/v4/public/components/discord" | |||
| _ "github.com/redpanda-data/connect/v4/public/components/elasticsearch" | |||
| _ "github.com/redpanda-data/connect/v4/public/components/elasticsearchv8" | |||
There was a problem hiding this comment.
nit: up to you, but I would just put this in the same elasticsearch package. You could also use a v8 subdirectory if you do want to separate the packages.
There was a problem hiding this comment.
I was going to keep them separate because I think we won't want to support the older elasticsearch output in cloud. I'll move it into a v8 subpackage like you suggested.
| action, err := batch.TryInterpolatedString(i, e.conf.actionStr) | ||
| if err != nil { | ||
| return fmt.Errorf("interpolating action: %w", err) | ||
| } |
There was a problem hiding this comment.
You should be using batch.InterpolationExecutor and reusing the executors across the batch to prevent N copies of the batch (TryInterpolatedString makes a shallow copy of the batch every time it's called).
|
PS: Before merging, would you mind adding a note to the Changelog about |
This new output uses the latest official Elasticsearch go library.
Using `batch.TryInterpolatedString` creates unnecessary shallow copies of the message batch, this new implementation should be more performant.
ooesili
force-pushed
the
elasticsearch-v8
branch
from
6475d37 to
2955575
Compare
This new output uses the latest official Elasticsearch Go library.
esutil.NewBulkIndexerbecause it has a it's own internal concurrency and flushing logic that we don't really need since the benthos engine provides that for us.createaction because it is not idempotent which I don't think would play well with the assumptions that Connect makes of it's plugins.