Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GitHub-Issue#2778: Added CloudWatchLogsSink #3083

Closed
wants to merge 50 commits into from
Closed

GitHub-Issue#2778: Added CloudWatchLogsSink #3083

wants to merge 50 commits into from

Commits on Jul 3, 2023

  1. Elasticsearch client implementation with pit and no context search (o… 

    …pensearch-project#2910)

Create Elasticsearch client, implement search and pit apis for ElasticsearchAccessor

Signed-off-by: Taylor Gray <[email protected]>
Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    @graytaylor0
    graytaylor0 authored and Marcos Gonzalez Mayedo committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    f4dc9da View commit details
    Browse the repository at this point in the history

  2. GitHub-Issue#2778: Refactoring config files for CloudWatchLogs Sink (#4) 

    Added Config Files for CloudWatchLogs Sink.

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    @MaGonzalMayedo
    MaGonzalMayedo authored and Marcos Gonzalez Mayedo committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    4387b29 View commit details
    Browse the repository at this point in the history

  3. Added fixes from comments to code (including pathing and nomenclature… 

    … syntax)

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    f1c25bb View commit details
    Browse the repository at this point in the history

  4. Refactoring config (#5) 

    Added default params for back_off and log_send_interval alongside test cases for ThresholdConfig.

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    @MaGonzalMayedo
    MaGonzalMayedo authored and Marcos Gonzalez Mayedo committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    9d640e5 View commit details
    Browse the repository at this point in the history

  5. Fixed deleted AwsConfig file 

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    b100ee3 View commit details
    Browse the repository at this point in the history

  6. Removed the s3 dependency from build.gradle, replaced the AwsAuth.. w… 

    …ith AwsConfig.

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    35859b0 View commit details
    Browse the repository at this point in the history

  7. Added modifiable back_off_timer, added threshold test for back_off_ti… 

    …mer and params to AwsConfig

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    7040186 View commit details
    Browse the repository at this point in the history

  8. Added fixes to gradle file, added tests to AwsConfig, and used Reflec… 

    …tive mapping to tests CwlSink

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    6b13e21 View commit details
    Browse the repository at this point in the history

  9. Added default value test to ThresholdConfig and renamed getter for ma… 

    …xRequestSize

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    3bb125a View commit details
    Browse the repository at this point in the history

  10. Removed unnecessary imports 

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    e5ee1e5 View commit details
    Browse the repository at this point in the history

  11. Added cloudwatch-logs to settings.gradle 

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    e90b05a View commit details
    Browse the repository at this point in the history

  12. Merge branch 'main' into main 

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    @MaGonzalMayedo
    MaGonzalMayedo authored Jul 3, 2023
    Configuration menu
    Copy the full SHA
    43a0a40 View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2023

  1. Added a quick fix to the back_off_time range 

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 4, 2023
    Configuration menu
    Copy the full SHA
    40cb280 View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2023

  1. Merge branch 'opensearch-project:main' into main

    @MaGonzalMayedo
    MaGonzalMayedo authored Jul 5, 2023
    Configuration menu
    Copy the full SHA
    1aac686 View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2023

  1. Merge branch 'opensearch-project:main' into main

    @MaGonzalMayedo
    MaGonzalMayedo authored Jul 7, 2023
    Configuration menu
    Copy the full SHA
    903ea26 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2023

  1. Merge branch 'opensearch-project:main' into main

    @MaGonzalMayedo
    MaGonzalMayedo authored Jul 12, 2023
    Configuration menu
    Copy the full SHA
    ffe5cbe View commit details
    Browse the repository at this point in the history

  2. Added Buffer classes, ClientFactory similar to S3, and ThresholdCheck 

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    b56a845 View commit details
    Browse the repository at this point in the history

  3. Removed unnecessary default method from ClientFactory 

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    b06ed0b View commit details
    Browse the repository at this point in the history

  4. Added comments in Buffer Interface, change some default values to sui… 

    …t the plugin use case more

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    7f5a432 View commit details
    Browse the repository at this point in the history

  5. Removed unused imports 

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    4576899 View commit details
    Browse the repository at this point in the history

  6. Changed the unused imports, made parameters final in the ThresholdCheck 

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    a539833 View commit details
    Browse the repository at this point in the history

  7. Made changes to the tests and the method signatures in ThresholdCheck… 

    …, made fixes to gradle file to include catalog

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    c89ea17 View commit details
    Browse the repository at this point in the history

  8. Removed unused methods/comments 

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    063e1d3 View commit details
    Browse the repository at this point in the history

  9. Added CloudWatchLogsService, CloudWatchLogsServiceTest and Retransmis… 

    …sionLimitException

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    1aad0b5 View commit details
    Browse the repository at this point in the history

  10. Fixed retransmission logging fixed value 

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    cf1f8e1 View commit details
    Browse the repository at this point in the history

  11. Fixed unused imports 

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    77f6d0f View commit details
    Browse the repository at this point in the history

  12. Fixed making ThresholdCheck public 

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    75d90fe View commit details
    Browse the repository at this point in the history

  13. Added fixes to ThresholdCheck and CloudWatchLogsService to decouple m… 

    …ethods

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    5f2f511 View commit details
    Browse the repository at this point in the history

  14. Fixed syntax start import in CloudWatchLogsServiceTest 

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    fdc5b00 View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2023

  1. Extracted LogPusher and SinkStopWatch classes for code cleanup. Addde… 

    …d fixes to variables and retry logic for InterruptExceptions

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    0399694 View commit details
    Browse the repository at this point in the history

  2. Changed method uses in CloudWatchLogsService and removed logging the … 

    …batch size in LogPusher

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    3c02e1d View commit details
    Browse the repository at this point in the history

Commits on Jul 20, 2023

  1. Added Multithreaded CloudWatchLogsDispatcher for handling various asy… 

    …nc calls to perform PLE's

and added tests

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 20, 2023
    Configuration menu
    Copy the full SHA
    c2a02ec View commit details
    Browse the repository at this point in the history

  2. Added fixesto test and defaulted the parameters in the config to Clou… 

    …dWatchLogs limits, customer can change this in config file

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 20, 2023
    Configuration menu
    Copy the full SHA
    abec5e3 View commit details
    Browse the repository at this point in the history

  3. Added exponential backofftime 

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 20, 2023
    Configuration menu
    Copy the full SHA
    9bbfedd View commit details
    Browse the repository at this point in the history

  4. Fixed unused imports 

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 20, 2023
    Configuration menu
    Copy the full SHA
    6e28adc View commit details
    Browse the repository at this point in the history

  5. Fixed up deepcopy of arraylist for service workers in CloudWatchLogsS… 

    …ervice, and fixed Log calling methods

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 20, 2023
    Configuration menu
    Copy the full SHA
    90418aa View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2023

  1. Merge branch 'opensearch-project:main' into main

    @MaGonzalMayedo
    MaGonzalMayedo authored Jul 25, 2023
    Configuration menu
    Copy the full SHA
    394bed7 View commit details
    Browse the repository at this point in the history

  2. Added CloudWatchLogsDispatcher builder pattern, fixed tests for Servi… 

    …ce and Dispatcher and modified backOffTimeBase

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 25, 2023
    Configuration menu
    Copy the full SHA
    5971190 View commit details
    Browse the repository at this point in the history

  3. Removed unused imports 

    Signed-off-by:Marcos Gonzalez Mayedo <[email protected]>
Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 25, 2023
    Configuration menu
    Copy the full SHA
    53086c4 View commit details
    Browse the repository at this point in the history

  4. Added resetBuffer method, removed unnecessary RetransmissionException… 

    …, and added logString pass in parameter for staging log events.

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 25, 2023
    Configuration menu
    Copy the full SHA
    26f18a1 View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2023

  1. Started making changes to the tests to implement the new class struct… 

    …ure (performance enhancement)

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 26, 2023
    Configuration menu
    Copy the full SHA
    a5b8be7 View commit details
    Browse the repository at this point in the history

Commits on Jul 27, 2023

  1. Refactored the CloudWatchLogsDispatcher into two classes with the add… 

    …ition of Uploader, introduced simple multithread tests for CloudWatchLogsService

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    070beef View commit details
    Browse the repository at this point in the history

  2. Fixed issues with locking in try block and added final multithreaded … 

    …tests to the CloudWatchLogsService class

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    6130b08 View commit details
    Browse the repository at this point in the history

Commits on Jul 28, 2023

  1. Added CloudWatchLogsMetricsTest, changed upper back off time bound an… 

    …d scale, and refactoring changes for better code syntax (renaming, refactoring methods for conciseness, etc...)

Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    69320ec View commit details
    Browse the repository at this point in the history

  2. Added changes to javadoc 

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    993cbd0 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'adding_cloud_watch_logs_service' into cloudwatchlogs_si… 

    …nk_implementation
    Marcos Gonzalez Mayedo committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    377a0fe View commit details
    Browse the repository at this point in the history

  4. Update data-prepper-plugins/cloudwatch-logs/src/main/java/org/opensea… 

    …rch/dataprepper/plugins/sink/client/CloudWatchLogsDispatcher.java

Co-authored-by: Mark Kuhn <[email protected]>
Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    @MaGonzalMayedo
    MaGonzalMayedo and Mark Kuhn authored Jul 28, 2023
    Configuration menu
    Copy the full SHA
    2b18115 View commit details
    Browse the repository at this point in the history

  5. Fixed comment on CloudWatchLogsDispatcher 

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    eee6197 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'adding_cloud_watch_logs_service' into cloudwatchlogs_si… 

    …nk_implementation
    Marcos Gonzalez Mayedo committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    e63e67d View commit details
    Browse the repository at this point in the history

  7. Added CloudWatchLogsSink and CloudWatchLogsSinkTest 

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>
    Marcos Gonzalez Mayedo committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    de0621f View commit details
    Browse the repository at this point in the history