-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(kubernetes source)!: Use Pod's UID
to exclude Vector's logs
#2188
Closed
Closed
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I know I asked this before but is this env var always available? If it is lets make sure we document this very clearly, if not we need to find a way to work around this and that may be just ignoring this and letting vector logs come through.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also curious.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is present if
.yaml
is configured with, and if it isn't configured like that, I wouldn't consider it a proper configuration. In the same way it isn't a proper configuration to run vector without mapping the folders necessary for it's operation.
Passing the logs would make it easy for us
brick(EDIT: to strong of a word, the Github runners became unresponsive so they were stopped automatically after some time) the node, as it happened with Github's nodes, if we accidentally, or forget, add a info log somewhere that depends on the number of logs passing. And even worse if the size of the info depends on the size of the log.For other ways, we filtered out logs for containers which had names starting with vector, which could be suprising for users that name their containers that way, but we could name vector container in some unique way, but then we are back at the beginning where we need the
.yaml
to be configured in a proper way.There is also another way, that just poped into my head, we could with intention log a message with unique string of characters and once we detect that string as a message in the log we would know which UID is ours and could filter them out, although file source would still unecessary be picking them up from the log file.
What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not really happy with this required env var personally. I think its a pretty bad UX hazard.
Do we know what other solutions do here? What does fluentd do?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For example this daemonset seems quite simple compared to ours https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/fluentd-daemonset-elasticsearch.yaml
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes! I am very much in favor of anything that improves reliability and confidence for v1 of our Kubernetes integration. I do not want to compromise that supporting old versions right now. If requiring >= 1.14 achieves that then let's do it.
That's ok. EKS also offers 1.14 and 1.15.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree, simplifying the source would be great. It seems like 1.14 is over a year old and technically not supported anymore (see here and here).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll add my 2 cents, we've been using in production for quite some time fluentd and there has been an attempt to use fluent-bit, and now we're replacing fluent-bit with vector. In all our configurations we've been deploying the entire log-related stack in its own namespace. Mostly because with an appropriated Role and PSP it can be easily restricted.
Even with the current (0.8.2) kubernetes source it works perfectly fine (because you can just whitelist any namespace except vector's).
Just saying that it can be a suggested pattern of deployment and in this case the solution can be less invasive. Otherwise filtering using Pod's UID can be opted-in.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Alexx-G we really appreciate your feedback! So if I understand correctly for you all it would make sense to have some way in the vector config to set the namespace you are deploying your log stack in. From this we can ignore all logs from that namespace? This to me seems like the ideal solution.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using a separate namespace is a particular deployment pattern, and not everyone can use that. So we can't solely rely on that - it would be a very significant limiting factor.
As an alternative to env vars, we could use downward API mounts: https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/
We can also communicate with the cluster API to determine the current container info: https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-api/#accessing-the-api-from-within-a-pod
Also, a
hostname
in the container environment is by default set to the pod name.That said, I like the explicit configuration as this PR proposes. This, being explicit, is much easier to understand and maintain for infra teams. I.e. when they operate on a configuration for vector - it becomes very simple to reason which logs are ignored and which are picked up. And this is, arguably, a better tradeoff when we're talking k8s.
Also, since the industry standard in k8s-compatible apps is to ship the deployment configuration as part of the release, I don't see this configuration detail as particularly problematic. We should provide a helm chart for our stuff anyway, as well as applicable daemonset/sidecar deployment files. This would be a proper implementation of our "good defaults" paradigm in the k8s sense.