Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

acquisition/victorialogs: add new datasource #3310

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

zekker6
Copy link

@zekker6 zekker6 commented Oct 30, 2024

Related issue: #3307

Data source supports:

  • cat mode with automatic adjustment of poll interval (same as one at Loki datasource)
  • tail mode by using tailing API

Most of the logic is exactly the same as Loki datasource.
The only difference is related to adding support of live dailing and different response formats.
VictoriaLogs uses newline delimeted JSON for both reqular query and tail modes.

Tested locally with the following config:

source: victorialogs
log_level: debug
url: https://vlogs-endpoint
mode: tail
limit: 1000
query: |
  app_name:nginx
labels:
  type: nginx
---
source: victorialogs
log_level: debug
url: https://vlogs-endpoint
mode: cat
since: 1m
limit: 1000
query: |
  app_name:nginx
labels:
  type: nginx

Data source supports:
- cat mode with automatic adjustment of poll interval (same as one at Loki datasource)
- tail mode by using tailing API
Copy link

@zekker6: There are no 'kind' label on this PR. You need a 'kind' label to generate the release automatically.

  • /kind feature
  • /kind enhancement
  • /kind refactoring
  • /kind fix
  • /kind chore
  • /kind dependencies
Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

Copy link

@zekker6: There are no area labels on this PR. You can add as many areas as you see fit.

  • /area agent
  • /area local-api
  • /area cscli
  • /area appsec
  • /area security
  • /area configuration
Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

Copy link

codecov bot commented Oct 30, 2024

Codecov Report

Attention: Patch coverage is 0.20408% with 489 lines in your changes missing coverage. Please review.

Project coverage is 40.83%. Comparing base (97c1f60) to head (a868de6).
Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
...odules/victorialogs/internal/vlclient/vl_client.go 0.00% 244 Missing ⚠️
...g/acquisition/modules/victorialogs/victorialogs.go 0.00% 244 Missing ⚠️
pkg/acquisition/victorialogs.go 50.00% 0 Missing and 1 partial ⚠️

❗ There is a different number of reports uploaded between BASE (97c1f60) and HEAD (a868de6). Click for more details.

HEAD has 2 uploads less than BASE
Flag BASE (97c1f60) HEAD (a868de6)
unit-windows 1 0
unit-linux 1 0
Additional details and impacted files
@@             Coverage Diff             @@
##           master    #3310       +/-   ##
===========================================
- Coverage   58.23%   40.83%   -17.40%     
===========================================
  Files         335      334        -1     
  Lines       37397    36488      -909     
===========================================
- Hits        21777    14899     -6878     
- Misses      13724    20108     +6384     
+ Partials     1896     1481      -415     
Flag Coverage Δ
bats 40.83% <0.20%> (-0.58%) ⬇️
unit-linux ?
unit-windows ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

"datasource_s3": false,
"datasource_syslog": false,
"datasource_wineventlog": false,
"datasource_victorialogs": false,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be listed in the Makefile too -- the only reason to "register" datasources is to optionally exclude from the builds with a tag, and still list them in "cscsli version"

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added this to Makefile in f4603e1, thanks for noticing

@zekker6 zekker6 requested a review from mmetc November 1, 2024 14:37
@buixor buixor added this to the 1.6.5 milestone Nov 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants