[IPFIX] Fix parsing when using buffered (TCP) input #194
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💚 CLA has been signed |
Abazigal
force-pushed
the
issue_tcp_ipfix
branch
from
bf3f262 to
f9e4069
Compare
Abazigal
changed the title
Abazigal
force-pushed
the
issue_tcp_ipfix
branch
12 times, most recently
from
ce564ef to
6d345b5
Compare
When using a TCP input, packets' data are buffered before logstash tries do decode them. Therefore, our decode() function will receive chunks of "random" sizes, that might contain 2 PDUs, 3.4 PDUs, etc. The current code parses only one PDU and discards the rest of the payload. Therefore, we can easily miss a PDU, and the next call will most likely parse the middle of a PDU, which will result in an error. The file ipfix.dat used during CI is actually a good example : it contains 3 IPFIX messages. But so far, the code is only considering the first one, hence the 7 flows returned instead of the 13 that the file contains. This commit makes sure each call consumes all the PDUs available in the payload, and the remaining data (beginning of another PDU) are buffered to be reused in the next call.
|
The remaining CI issue comes from master. |
Abazigal
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When using a TCP input, packets' data are buffered before logstash
tries do decode them. Therefore, our decode() function will receive
chunks of "random" sizes, that might contain 2 PDUs, 3.4 PDUs, etc.
The current code parses only one PDU and discards the rest of the
payload. Therefore, we can easily miss a PDU, and the next call will
most likely parse the middle of a PDU, which will result in an error.
The file ipfix.dat used during CI is actually a good example : it
contains 3 IPFIX messages. But so far, the code is only considering
the first one, hence the 7 flows returned instead of the 13 that the
file contains.
This commit makes sure each call consumes all the PDUs available in
the payload, and the remaining data (beginning of another PDU) are
buffered to be reused in the next call.