Feat/batch templating #347
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
colmsnowplow
force-pushed
the
feat/batch-templating
branch
from
321429f to
a80ea8d
Compare
colmsnowplow
force-pushed
the
feat/batch-templating
branch
from
01a5aba to
4631cc7
Compare
istreeter
approved these changes
Jul 3, 2024
Comment on lines
+349
to
+351
| for _, msg := range goodMsgs { | ||
| msg.TimeRequestStarted = requestStarted | ||
| msg.TimeRequestFinished = requestFinished | ||
| } |
There was a problem hiding this comment.
Interesting you do this even if err != nil
There was a problem hiding this comment.
It's most valuable if err != nil! We introduced it on request from Tech Ops, because it tells us when lag/issues are most likely external :)
colmsnowplow
force-pushed
the
feat/batch-templating
branch
from
4631cc7 to
853d9aa
Compare
provideRequestBody can't be compatible with plain strings and play nicely with our data. This commit changes it to expect a JSON - if there is some use case for plain strings it can still be done via a transformation + a template.
Won't be exact but will err on the side of caution since the template file's data will be _slightly_ bigger than the empty template.
colmsnowplow
force-pushed
the
feat/batch-templating
branch
from
853d9aa to
07dc64e
Compare
colmsnowplow
added a commit
that referenced
this pull request
Sep 11, 2024
Refactors the HTTP target to handle batches of data in one go, adds the ability to provide a template to get those batches of data into a request body, and adds a helper function to cover the case where there is no template provided. This involves breaking changes - the previous behaviour can be achieved by configuring a template if the data is already JSON, or a transformation + a template if not - but we don't believe there are any use cases requiring us to do so. - Refactors the http target to handle batches - Adds configuration of batch sizes for http target - Adds a templater function - Adds a helper function which creates a JSON array from the data where no template is provided - Adds a function to group data by dynamic headers where that is configured - to ensure that this feature still works - Adds tests for these features These features also introduce a new possibility for failure in the HTTP target. The http target now expects valid JSON (previously it could be anything) - where we hit an error in either templating this data - or in the case of no template in transforming the data into a JSON array request body - these messages are treated as invalid, and will be sent to the failure target. --------- Co-authored-by: Piotr Poniedziałek <[email protected]>
pondzix
added a commit
that referenced
this pull request
Oct 25, 2024
Refactors the HTTP target to handle batches of data in one go, adds the ability to provide a template to get those batches of data into a request body, and adds a helper function to cover the case where there is no template provided. This involves breaking changes - the previous behaviour can be achieved by configuring a template if the data is already JSON, or a transformation + a template if not - but we don't believe there are any use cases requiring us to do so. - Refactors the http target to handle batches - Adds configuration of batch sizes for http target - Adds a templater function - Adds a helper function which creates a JSON array from the data where no template is provided - Adds a function to group data by dynamic headers where that is configured - to ensure that this feature still works - Adds tests for these features These features also introduce a new possibility for failure in the HTTP target. The http target now expects valid JSON (previously it could be anything) - where we hit an error in either templating this data - or in the case of no template in transforming the data into a JSON array request body - these messages are treated as invalid, and will be sent to the failure target. --------- Co-authored-by: Piotr Poniedziałek <[email protected]>
pondzix
added a commit
that referenced
this pull request
Nov 6, 2024
Refactors the HTTP target to handle batches of data in one go, adds the ability to provide a template to get those batches of data into a request body, and adds a helper function to cover the case where there is no template provided. This involves breaking changes - the previous behaviour can be achieved by configuring a template if the data is already JSON, or a transformation + a template if not - but we don't believe there are any use cases requiring us to do so. - Refactors the http target to handle batches - Adds configuration of batch sizes for http target - Adds a templater function - Adds a helper function which creates a JSON array from the data where no template is provided - Adds a function to group data by dynamic headers where that is configured - to ensure that this feature still works - Adds tests for these features These features also introduce a new possibility for failure in the HTTP target. The http target now expects valid JSON (previously it could be anything) - where we hit an error in either templating this data - or in the case of no template in transforming the data into a JSON array request body - these messages are treated as invalid, and will be sent to the failure target. --------- Co-authored-by: Piotr Poniedziałek <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refactors the HTTP target to handle batches of data in one go, adds the ability to provide a template to get those batches of data into a request body, and adds a helper function to cover the case where there is no template provided.
This involves breaking changes - the previous behaviour can be achieved by configuring a template if the data is already JSON, or a transformation + a template if not - but we don't believe there are any use cases requiring us to do so.
These features also introduce a new possibility for failure in the HTTP target. The http target now expects valid JSON (previously it could be anything) - where we hit an error in either templating this data - or in the case of no template in transforming the data into a JSON array request body - these messages are treated as invalid, and will be sent to the failure target.