Commits on Oct 17, 2024

1. Bugfix for BufferedTokenizer to completely consume lines in case of l… …

   …ines bigger then sizeLimit (#16482)
   
   Fixes the behaviour of the tokenizer to be able to work properly when buffer full conditions are met.
   
   Updates BufferedTokenizerExt so that can accumulate token fragments coming from different data segments. When a "buffer full" condition is matched, it record this state in a local field so that on next data segment it can consume all the token fragments till the next token delimiter.
   Updated the accumulation variable from RubyArray containing strings to a StringBuilder which contains the head token, plus the remaining token fragments are stored in the input array.
   Furthermore it translates the `buftok_spec` tests into JUnit tests.
   
   (cherry picked from commit 85493ce)

   

   andsel authored and logstashmachine committed Oct 17, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 0b56323
   • Browse repository at this point

   Copy the full SHA

   0b56323 View commit details

   Browse the repository at this point in the history